Fads come and go. Popularity rises and falls.
Games of all types are not immune to this phenomenon. Some games probably begin their popularity streak due to traditional marketing, but most games become popular through social networking or through interest transference.
According to the USBF web site, Bridge became popular at the end of the 18th century, but it's popularity did not initially result in a decline in popularity of Whist or other games. It was only in the late 1920's, when next generation rules about auctioning were introduced and solidified in Bridge that Whist's popularity began to decline.
While arcade games in the 1980's had to compete with each other for space on the game floor, the games' popularities suffered more as a result of later produced games than as a result of direct competition with contemporarily produced games. I don't think that Asteroids became less popular because of Pac Man. Some people still enjoyed playing Asteroids and Pac Man at the end of the 1980's but the vast majority had gone on to next generation games.
My hypothesis is that competition from same generation gaming is an order of magnitude less than the competition from next generation gaming.
In board gaming, war games were "the thing" in the 1980's. These declined as a result of the rise of video and computer games, especially the computer games that did the war game thing so much better, such as Age of Empires and Civilization.
The early 1990s was a quiet time for games until Magic the Gathering burst onto the scene in 1992-93. A whole generation of CCGs thereafter did nothing to erode Magic's market, even while some were probably better. These games fed into each other. The odds are that you only played a CCG after playing Magic (Kuma notes the same thing about D&D and RPG players). D&D hasn't been displaced, because a) it reinvented itself with a next generation version, and b) other RPGs are still the same generation as D&D - playing catchup and imitate. They may be better than D&D; that may give them room to grow. But it is unlikely to reduce D&D's popularity all that much as a result.
In the late 1990's Eurogames hit the scene with Settlers of Catan and follow-ups. Each new and better Eurogame since Settlers is doing nothing to damaging Settlers' popularity. With all the fancy new arrangements of mechanics and so on, they do not represent the next generation of games.
Instead, the games feed each other business. Sure, sometimes a person can only buy a single game and has to make a choice about which one. And there are a lot of mediocre games that are popular for a month or two and then sink away. But by and large, the games that achieve popularity continue to do so. Until.
I hesitate to continue this line of thought, but it's not unnatural. And any game company worth the business cards its name is printed on has to be thinking the same thing.
What is next? You can comfortably continue to make more auction / trading / tile laying / negotiation / action point / resource building games, but only until something really new and radically better comes along. Then your market share is going to start dwindling. A small group will continue to play the old games, but that will be barely enough to sustain a business.
It could be a new game that really uses all of these mechanics so well that it becomes a next generation all by itself. But that is doubtful. More likely, we're going to find something new in board gaming.
Frankly, it's high time that we did. To find new ideas we have to look outside of the myopic board game world of today and incorporate ideas from other gaming streams. For instance, Chris Bateman has a great article about new thoughts in computer games, which could be very relevant as a springboard for board game design.
There is a huge potential cross-pollination from computer game design to board game design and vice versa that needs to take place. And by that I don't mean simply trying to represent board games on a computer or vice versa.
I wish Eurogames well, but the first generation of Eurogames is now more than ten years old. That's a long time in game years. How much longer will this continue?
P.S. Another recent article by Craig Perko talks about the difference between multi-player and single-player computer game design, which is also relevant to board game design.