An earthquake measuring 7.2 on the Richter scale was recorded late Saturday with its epicentre in the sea south of Mindanao island, about 500 kilometres (313 miles) from Dent Haven near Lahad Datu on Sabah's east coast.
The Prime Minister's Crisis and Disaster Centre issued Malaysia's first-ever tsunami alert after the meteorological department told it about the quake.
Centre director Muhamad Muda said parts of Sabah, especially the state capital Kota Kinabalu, Tawau and several coastal areas, felt the tremor.
"Our department immediately sent out the alert to the Malaysian Control Centre which relayed the message to the Sabah police, military, the information ministry and the local meteorological department," he said.
Police urged locals in some areas to evacuate coastal homes.
Several seaside resorts on Sabah's west coast immediately alerted their guests. News organisations in Sabah were flooded with calls while people in Tuaran near Kota Kinabalu were caught in traffic jams as they fled towards Mount Kinabalu.
Police were seen in many places calming the public after midnight as they refused to leave mosques and other higher ground.
Even after the alert was scaled down, some Sabahans were still jumpy. "We were not taking any chances," Mohamad Affendi, 44, told AFP, as he made his way to the hills.
Sabah Chief Minister Musa Aman said Sunday the panic demonstrated the need for the federal government to instal an early warning system off Sabah.
Science, Technology and Innovation Minister Jamaludin Jarjis said Malaysia would install a tsunami early warning system in northern peninsular Malaysia and in waters off Sabah and Sarawak states on Borneo by year-end.
Malaysia was holding talks with US maritime experts about installing the five million dollar system, the official Bernama news agency quoted him as saying.
Brunei on Borneo island was keen to share the system with Malaysia, he said. "God willing, we will have have the system in place by year-end."
A total of 68 people died in Malaysia, mostly in the northern resort island of Penang, following the December 26 tsunamis which killed more than 294,000 people across the Indian Ocean.