Yes, a paint job should last longer than three years. However, exterior paint requires regular maintenance, (especially in areas with volatile weather patterns) to maintain its appearance and extend its life.
The mildew growth you are experiencing is not uncommon, particularly on newer homes. Builders frequently use paints which do not contain a mildewcide.
Your paint may be in fine condition,. I would recommend cleaning the exterior of your house, and then inspecting the paint. In addition to being unsightly, mildew can also destroy the paint film. Regularly cleaning the exterior will not only improve the appearance but extend the life of your paint as well