In DP8, go to the Digital Performer menu>Preferences.
In the General category choose Audio Plugins.
DP8 automatically scans and enables plugins in the Primary External Plugin Format.
If VST is the primary format and you want to use an Audio Unit plugin, enable it from the plugin set by clicking the box next to the AU plugin you want enabled.
If AU is the primary format and you want a VST plugin, enable the box next to that plugin.