Drinks with a few old buddies in recent weeks yielded a surprising epiphany for Fidelity Investments' Thomas Soviero: It was time to lighten up on his holdings of retail stocks.

His pals had lamented the added burden of rising gas prices and increased taxes, noting that making ends meet was becoming increasingly difficult. More than a couple said they were cutting back on dining out, vacations and other spending.

'I have many friends in the blue-collar world, and the payroll tax and gas prices hit people harder than I expected,' said the 49-year-old Boston native, who manages $6.7 billion for Fidelity. He says he is now 'underweight' retail stocks compared with his benchmark index, after starting the year in line.

Across Wall Street, many investors are coming to a similar conclusion.

It is a shift in thinking. Consumers have been resilient during the past few years, despite slow economic growth. As 2013 began, many investors were ratcheting up expectations for the economy after the worst of the fiscal-cliff spending cuts were avoided. With the job market slowly improving and housing stabilizing, many thought that while the economy wasn't going to boom this year, it might be able to achieve a higher level of growth.

But now investors are rethinking that scenario. They worry that the new burdens may soon take a toll, prompting Americans to pare back on nonessentials and seek out cheaper versions of staple items.

Coming into this year, Mr. Soviero held retail and consumer stocks such as videogame retailer GameStop, beauty retailer Sally Beauty Holdings, cable giant Comcast and cinema operator Cinemark Holdings .

He has reduced some of those retail holdings. He remains optimistic about domestic auto makers