The mother of the children was not at home when Wednesday's fire broke out in the sixth-floor apartment of the Crown Heights residence at about 12:30 p.m., authorities said.
Firefighters were on the scene minutes later and had it under control by 1 p.m. The fire department said the blaze apparently started in the bedroom.
Killed in the blaze were 1-year-old Sebastian Jean, 2-year-old Fabrice Jean, and 42-year-old Myrtel Jean, believed to be their father, police said.
Fire investigators said it appears the man may have attempted to put the fire out by himself.
Horrified neighbors watched as firefighters carried out the victims, who later died at a local hospital.
"When they brought the first child out, the child was limp — it looked like a rag," Chaquama Carr, 34, told the Daily News.
Uniformed Firefighters Association President Steve Cassidy released a statement claiming precious minutes were wasted when the city's new dispatch system initially sent firefighters to the wrong address.
Fire department spokesman Frank Gribbon denied that claim, saying firefighters responded to the vicinity of an intersection given by 911 callers.
"They did not respond to the wrong address," Gribbon said. "They responded to the location provided by both calls to 911."
Gribbon said firefighters arrived at that location in just over three minutes, learned that the fire was around the corner and immediately began trying to rescue the victims and put out the fire.
The firefighters union has been critical of the new dispatch system, which was put in place over union objections.
Earlier this month an emergency dispatcher's keystroke error sent fire trucks toward the wrong address during a blaze that killed three people in Queens.
The mistake was quickly corrected while the trucks were en route, but it was not clear whether it was a factor in the deaths.