The man died after being thrown overboard from a dinghy as he tried to secure a yacht at its moorings at the upmarket suburb of Mosman on Sydney Harbour, police said.
Despite the efforts of water police, paramedics and a member of the public, the man, who is yet to be identified, died in hospital a short time later.
Daily ferry services which carry thousands of passengers across the harbour to the central business district were cancelled, roads and railway lines were flooded and flights into and out of Sydney airport were delayed.
A weather bureau spokesman said part of the city had recorded its wettest September day since 1883 with a 107.2mm drenching, while some areas exceeded their average rainfall for the entire month of September.
The NSW State Emergency Service received more than 500 calls, mostly seeking help with fallen trees, flash flooding and leaking roofs.
The wild weather was caused by a low pressure system crossing the state of New South Wales late Wednesday, which then deepened off the coast of Sydney Thursday.
Overnight storms brought welcome rain to most of New South Wales, although some regions, desperately dry after years of drought, received only 1mm.
Residents living to the north of Sydney awoke to a blanket of snow Thursday morning, the first major snowfall of the year.