Fishers Island is the Anti-Hampton

by Ben Diamond Photographed by Ben Diamond
Monday, July 2, 2018

Ask a resident of Fishers Island for his thoughts on the stock market or on the country’s best golf courses, and he’ll talk your ear off. But ask about the island and what the Sicilians call omertà takes over. “They wouldn’t be pleased to see me quoted,” said one longtime Fisherite when reached for comment. More than one requested anonymity as a condition for conversation about the summer community.

“It’s such an unspoiled place, and everybody prefers that it remain that way. The less known it is, the better,” said a resident. “You can be on the beach by yourself on a holiday weekend and not see another soul. There’s not anywhere else on the East Coast, from Maine to Florida, where you can have so much beach to yourself.”

Fisherites come from the same stock as visitors to other Northeastern summer spots. “A lot of people come for July or August,” said Bunty Armstrong, the widow of long-time Whitney Museum director Thomas Armstrong. “They’re mostly from Connecticut and New York, but there are some Texans and some Bostonians.”

Where they differ is their uniquely cultlike devotion to the place. Many of the families that live there have done so for generations. Some, like the fashion editor Polly Allen Mellen, have been going since the 1930s. Their numbers include the Gosses of Connecticut (among them former CIA director Porter Goss), the Keans of New Jersey (former governor Tom Kean), the Du Ponts of Delaware, and a number of equally influential figures who have managed to escape the public eye entirely. Yes, while the WASP aristocracy that once ruled America has mostly disappeared from view, it lives on at Fishers Island.

Like the fictional village of Brigadoon, the island is a place that seems to exist out of time itself, where the conventions of the contemporary world do not hold. “The first time I came to the island, I felt like I was thrown back in time to my early childhood, when things felt different—smaller, slower, cleaner, safer,” said one longtime summer resident.

The island hasn’t been entirely immune to the passage of time. “The world has changed,” said one resident. “In earlier days, you would never have seen a gay couple enter a house here. But that happens now.”

But in the important ways, the island is the same place it’s always been. The herons, egrets and quails still come out when the weather gets warm. The private Hay Harbor and Fishers Island Clubs still offer world-class golf. And while the faces may have changed, the last names on the island remain the same.

And that’s no accident.

Part of that stems from location alone. Although nearer than other Old Guard enclaves like Bar Harbor or Watch Hill, Fishers Island is a real hassle to get to. Located eleven miles from the tip of Long Island at Orient Point and seven miles from New London, Connecticut, the island is far enough away that it can’t clearly be seen from either. Even after the three-hour drive to New London, it’s only halfway into a 45-minute ferry ride from the mainland that it even reveals itself. “It’s built into the geography of the place that it can never be that populated,” said one summer resident. Or, as the website for local realtor Shutters & Sails has it, “Fishers Island is the ultimate gated community—only the gate is a ferry.”

And though there have been proposals to connect Fishers Island to the mainland—usually as part of a bigger bridge between Long Island and Connecticut—for now, the ferry remains the usual way in. It’s all part of the exclusivity, making visitors feel like trespassers before they’ve even left the New London dock. “Are…are you staying with anyone?” the ticket agent will ask you if you’re unaccompanied, sure you’re engaged in subterfuge of some sort or, at best, very lost.

The ride that follows offers a good illustration of what kind of people frequent the place. On my own journey, I shared a bench with two older women carrying L.L. Bean totes and wearing fleece—one emblazoned with the logo of the Women’s Open Golf Championship. These ladies were the classic Fisherites, wrinkled but fit, rugged in a way that their Hamptons counterparts simply aren’t.

“Fishers is sort of the anti-Hamptons,” said one longtime summer resident. “Even going back several decades, it was the anti-Hamptons. And before that, it was an anti-Newport. Everything is much less formal here.”

This is a mark of pride among Fisherites—nearly all seem to have a pronounced, almost gleeful disdain for the creature comforts of other resort towns. “This island has very little to offer for people who want to go shopping, or who want to go ‘out’ at night, or go to restaurants,” said Polly Mellen.

Once you actually start to approach the island, the most notable thing is how little there is—no marina, no waterfront diner with a steering wheel on the wall, just a few small buildings, a desolate-seeming dock and lots and lots of rocks. Natural fortification like that served an important purpose once upon a time, stopping pirates from stealing the island’s whale blubber or something. Today they serve a different function. “Avast ye!” they seem to say. “This island has no hotel!”

There isn’t a lot to do on Fishers Island, either—or at least, not much for the unconnected visitor. That’s partly by design—places full of amenities and destinations tend to attract unwanted attention. “One reason,” the social arbiter Cleveland Amory wrote back in 1952, “that everyone does not know about Fishers Island and Hobe Sound [a similar resort, near Palm Beach] is that these islands do not wish to be known; in fact, their anonymity has been largely responsible for making them perhaps the most formidable social resorts still in existence.” Though observers might quibble about the island’s relative formidability today, Amory’s words still ring true.

“As [visitors] get off the ferry, there’s just so little for them to do,” said Polly Mellen. “They can ride around on their bicycles, and go to the town, which has just a few shops, and that’s all. There are no beaches for them to go to except a beach called South Beach, which is very stony, and they can go to the one restaurant, which is called the Pequot Inn, and that’s it.”

In fact, the island’s commercial prospects have only decreased in the past half-century. The movie theater present in Amory’s day (though even then, he wrote, there was “always a question each summer of whether or not it will be running”) is no more. Nor is there any hotel: the Pequot Inn, which until recently offered the island’s only public lodging, closed in 2011. (Though, as Mellen said, it continues to operate as a restaurant and bar—a small blessing for the bored children of initiates who return after reaching drinking age.)

But even without a dedicated commercial district, Fisherites have their routine. “Every year it’s kind of the same thing,” said one resident. “Fourth of July fireworks, a sandcastle contest, cocktail parties for adults and some philanthropic matters.”

“There were always cocktail parties when I was little,” remembered the writer Peter Heller, who from the late 1950s to the 1980s spent most summers on the island. “It was like a Cheever story—people were always showering after the beach, getting ready, and going off to cocktail parties.”

Although, as one resident added. “There are not many places to go except other people’s houses.”

Which is not entirely true. Those who have found full acceptance on the island can graduate to its most exclusive precincts: the clubs. Nearly all the island’s social life is concentrated around its two social clubs, the Hay Harbor Club and the Fishers Island Club. The Hay Harbor is on the island’s western end, not too far from the main road, while the Fishers Island Club is to the east, past a guarded gate that also protects some of the island’s homes (yet another exclusive hurdle!). The clubs offer golf, tennis, beach access and, most important, breakfast, lunch and dinner. “I would find it hard to be here were it not for the clubs,” said a resident. “Unless you have someone in your house who loves to cook or who is hired to cook. If you want to go out to eat, there is virtually nowhere to go. You could survive on the island without them, but it’s not something I would want to do myself. I’m not a good cook!”

Members take many of their meals at the clubs—one estimated that she eats a full 25 percent of the time at hers. And owing to this, it’s no wonder that the clubs are packed—both, in fact, having waiting lists to accommodate the heavy demand for membership.

With all that difficulty, why visit the island at all? There’s no real question there: it’s extremely beautiful, and uniquely so. Parts of it are like Long Island, others are like the coast of Maine, and some even feel like they’ve been airlifted from Waikiki. “Almost every house that you enter has an entirely different view,” said a resident. “There are so many twists and turns to the coastline that it’s like you’re on a different coast in a different part of the country wherever you go.”

That topography only enhances the island’s mystery. It has two main roads, with dozens of side roads that branch off before returning back. It’s easy to get lost, and easier still to drive the entire island without seeing anyone or anything.

“There’s been a constant fight to maintain the year-round population, which has been declining since the ’30s,” said Pierce Rafferty, the director of the island’s Henry L. Ferguson Museum (and Barbara Bush’s nephew). The island was slow to develop—as late as 1876, there were only seven houses, since it was almost entirely owned by the Winthrop family, and operated as an offshore farm run by lessees. Despite being one of the first settlements in North America, Rafferty says, “You had no population here until it opened up as a resort beginning in 1877. The first land was only sold on the island in 1876. Up until that point, not one foot of land on the island had been sold. It had all been privately held.”

Although in the 1920s there was a push to develop the island in the same way as other resort towns on Long Island and Connecticut had been, the Great Depression put a stop to that. The closure of Fort H.G. Wright in the late 1940s was similarly harmful, and precipitated a steady decline in population throughout the coming decades. “You went from a population in, say, 1960 of around 500 year-round to maybe 230 now,” says Rafferty.

“On the other hand,” he adds, “You have a greatly increased summer population. It varies tremendously, but there’s certainly at least two or three thousand people on the island—and on July Fourth weekend, up to four or five [thousand]. In 1960, on a regular summer weekend, that number would have been closer to 1,200 in summer.”

Of course, the island does encourage some visitors. There are the friends and relatives that come as guests. “I’m always bringing up friends, though some don’t turn into repeat visitors,” said one summer resident. “Some find the trip grueling.”

For the past six years, the island has also been the home of Lighthouse Works, a public art commission and artist-in-residence program not far from where the ferry lands. From a pool of 1,200 applicants, four are selected to spend six weeks on the island. The plan was, as executive director Nate Malinowski put it, “Let’s just invite some artists to the island and see what happens.” The four women in the program when I visited acknowledged it was isolating—“We’re alone a lot,” said one—but all agreed that the slow pace made them more aware of their surroundings, and put them in a more reflective mood. “We go to the Pequot, too,” another added.

And then there are the mainlanders. Though the island’s year-round population has diminished, Rafferty is quick to point out that that statistic is a bit misleading. “It’s not as though there are fewer people working on the island,” he says. “The shift has been that a lot of the work force comes in on commuter boats.” And indeed, there were plenty of laborers on their way to the island on my ferry, including an amiable African-American carpenter with whom I struck up a conversation. I asked him whether there was anything AVENUE readers should know about the island. “They got black people here,” he said. “Working.”

Owing in part to Fishers Island’s high property taxes, the public school is another magnet for visitors—students, specifically. Leave on the 4:45 ferry, and you’ll be surrounded by schoolchildren returning to the mainland.

Still, all of that can’t paper over Fishers Island’s biggest challenge: a place with no year-round population, sooner or later, will atrophy. And for the hardy folk who try to scrape out a living during the off-season, a real resentment seems to be brewing.

A representative from the island’s Oyster Farm, which otherwise declined to have any involvement with this story, made it clear: “We would rather have more working people here than vacationing, leisure-seeking people.”

While the older generations who once lived on the island may have been cloistered and private, they were polite to their less well-off neighbors in a way that simply isn’t the case today. In this regard, there is a similarity to enclaves like Sag Harbor—or at least, to the complaints about them.

“This place has really gone downhill,” said one local, a remnant of a less rarefied time in the island’s history. “All the old money’s gone, and it’s been replaced by all this new money.” He recalled that Henry Luce III had once held a party to which he invited all his society friends, as well as plumbers, electricians and other working-class people, who in those days still lived on the island.

“I couldn’t see that happening today,” he added.


Every Day We’re Trufflin’

Belmond Castello di Casole launches truffle-hunting experience


Postcards from Jackson Hole

AVENUE goes west to explore our National Parks.

In The Magazine
At Home

Fully Engaged

A Snapshot into the Life of Bonnie Lautenberg