Alibaba Group Holding Ltd., challenged by Beijing’s yearlong clampdown on private enterprise, is facing another problem: growing competition.

For more than 15 years, Alibaba was China’s unassailable e-commerce champion. The company founded by Jack Ma rose to become one of the world’s biggest and most valuable businesses, using hard-knuckle tactics to muscle out challengers.

As China’s e-commerce industry has matured, consumers have started to embrace new ways of shopping that favor browsing and interaction over targeted product searches. That trend has left Alibaba playing catch-up in some areas, and competitors have used the shift to gain a foothold in the world’s largest online retail market.

Alibaba remains the leading platform in online shopping, but its share of China’s retail e-commerce market has fallen to a projected 51% in 2021 from 78% in 2015, according to research firm eMarketer.

Making inroads against Alibaba have been rival Tencent Holdings Ltd., which is incorporating online stores into its ubiquitous WeChat social-messaging app; Pinduoduo Inc., a six-year-old e-commerce app that has injected gamelike elements into shopping and drawn in bargain hunters with lower-priced goods; and Douyin, TikTok’s sister app in China, which is selling products through short videos and live-streaming with the help of its algorithms.

Alibaba has responded by investing more in areas such as content creation, live-streaming and discount goods. In May, Chief Executive Daniel Zhang listed increased competition as one of the company’s biggest obstacles of the past year and said any profit that exceeded last year’s would be poured back into improving its e-commerce businesses.

“When we used to talk about marketplace-based core, it was Taobao and Tmall," Mr. Zhang said in August, referring to the company’s flagship shopping platforms. “But now we’re working hard on building all these different businesses, each with its own unique and compelling value proposition." When asked about its initiatives to address competition, Alibaba referred to its past news releases and earnings statements.

Alibaba’s revenue still dwarfs that of other players in Chinese e-commerce, and its enormous size continues to give it major advantages. Alibaba’s size, however, has put the company in the crosshairs of regulators, who have been going after China’s tech giants.

Earlier this year, Beijing’s top market regulator found that Alibaba had abused its market position to keep its merchants from selling on competing platforms. The company was fined a record $2.8 billion in April. At the time, Alibaba said it accepted the penalty and vowed compliance.

Still, analysts said the increasing pressure on Chinese tech companies to play fair could limit how quickly and aggressively Alibaba can react to new threats.

Alibaba’s newest rivals face their own trials. Pinduoduo isn’t profitable and has relied on subsidies for some of its growth. The majority of Douyin’s sales last year were completed through platforms run by such competitors as Alibaba or JD.com Inc. Pinduoduo and ByteDance Ltd., which operates Douyin, declined to comment.

WeChat is under regulatory pressure to open its huge social-media and messaging ecosystem, its greatest advantage in commanding traffic, to other e-commerce players. Tencent declined to comment.