Phillyrea. This is one of two species of this genus which are evergreen shrubs to small trees in the Olive family, Oleaceae, and very close relatives of Osmanthus. It is rarely available in American nurseries and rarely seen in the landscape. Architectural Trees, a nursery in the UK which grows specimens plants for the landscape comments: "This tree has rather aristocratic associations (we call it posh) – but only because of its long lineage and its widespread use in horticulture for many centuries. We often sell these to English Heritage and the National Trust when they’re restoring Tudor, Elizabethan or Mediaeval gardens. There’s masses of evidence to show that Phillyrea was used extensively hundreds of years ago. The diarist John Evelyn (a keen gardener in the 17th century and contemporary of Samuel Pepys) writes of them frequently – as if they were then as common as box, yew and holly. What happened? They used to be so widespread and now they’re rare. Garden historians are perplexed and so are we."
Someone brought a stem of this to me from the courthouse grounds in Camden, SC where they had seen it flower. I saw one other plant on the grounds of the University of Georgia at the Alumni Association House, I believe. Oregon State University describes it thus.: "Broadleaf evergreen tree/shrub, to 30 ft (9 m) tall, similar width, rounded shape. Leaves opposite, simple, ovoid, to 6 cm long, margin toothed, dark green. Flowers greenish-white, not showy, fragrant, in clusters. Fruit blue-black, spherical, 6 mm."
The plant in Camden was in full sun with an almost 12" trunk diameter; whereas, the one at UGA was in filtered sun to part shade. Both were handsome specimens. It can lend itself to a small tree or a large shrub which lends itself to shearing because of its smallish leaves. I have never seen any pests on it, nor has it been eaten by our deer population. This is one of those outstanding plants to baffle one of the better plant identifiers. The third picture is courtesy of Oregon State University and the fourth picture is courtesy of the University of British Columbia Botanical Garden.