On the Yahoo! DITA Users Group, Paul Masalsky suggested this approach to glossaries in HTML-based output.
1. Create glossary definitions as separate DITA topics (perhaps use the glossary specialization).
2. Instruct content authors to enter <xref>s to those glossary topics for the desired popups.
3. Content authors should specify <xref outputclass="popup"> for each <xref> that is to be a popup.
4. For all <xrefs> labeled with outputclass="popup", output the following instead of the typical <a name=> anchor that defines standard hyperlinks (the likely place is the setanametag template in dita2htmlimpl.xsl, although you probably want to call a custom module to enable DITA toolkit upgrades to occur more easily):
5. Then you'd have to write the InvokePopup JS routine, which sets a position and boundaries for the popup box. I'm going to scour the net for reusable JS for this first.