These questions always raise questions of comparability across sports. I'd be willing to stipulate that we could make the comparison across all events that are races against time or contests for distance (e.g., excluding events like diving, figure skating, and gymnastics where judges play such a critical role).
Among those events, and even if Phelps should go on to 3 more gold medals, I am still going to place Eric Heiden's performance in the 1980 Winter Games as the greatest of all time. Heiden won all 5 of the men's speed skating events, all of which were individual events, at distances ranging from 500 meters to 10,000 meters. By contrast, 3 of Phelps' events are relays, and he swims only in a narrow range of distances, 100 to 400 meters. His Olympic performances are amazing to watch, both in Athens and Beijing, but I think Heiden was without equal.