Olive GardenWiki CommonsOlive Garden is one of the restaurants that has been hurt by a decline in casual dining.
Americans are abandoning restaurants, with financial concerns named as the culprit. 
The average American goes out to dinner 74 times a year, The Washington Post's Wonkblog reports, citing consumer research firm NPD group. That's the lowest reading in three decades. 
The popularity of take-out and prepared foods, like frozen pizzas, has partially contributed to the decline, reports Roberto Ferdman at Wonkblog.
But Americans are also worried about cash and looking to cut back.
"Consumers spent roughly 7 percent less on eating out in 2010, compared to the year before, according to the Bureua of Labor Statistics," Ferdman writes. "Meanwhile, people are buying more groceries, because it's cheaper."
The Federal Reserve Board reports that all groups of consumers had a lower mean income in 2013 than they did in 2007. Mean wealth also declined for all the groups.
The trend is hurting business at casual dining establishments like Olive Garden, Applebee's, and Red Lobster. 
The restaurants have tried implementing more promotions to lure back customers. 
Even McDonald's customers are complaining that the fast food fare is too expensive, according to a Bloomberg report. 
The retail sector is also hurting. 
Low-income and middle class retailers including Wal-Mart, JCPenney, Target, Macy's, and Family Dollar have reported disappointing earnings this year
Executives at these retailers say that Americans are increasingly unwilling to spend money on discretionary items, despite modest gains in the job market. 
"Low-end and middle-income retailers are still suffering because people are buying so close to need," Brian Yarbrough, consumer analyst at Edward Jones, told Business Insider. "In the past, retailers could depend on people spending a little more."