I didn't know this before today. I found this information on Collagitation. Easter is the only holiday in the christian calendar, where they forgot to change the pagan reference when they booted out the old gods and took it over. Its name still bears homage to the goddess, Oestre. She was, of course, a fertility goddess, hence all the rabbits and eggs.
I've always thought the eggs symbolized the new beginning or breaking through, something like that. Now it all makes sense.