ralph.m has it right. I run into this problem on a daily basis at work. Our partner's all use jQuery, and we use Prototype. It's a simple fix really.
At the end of your jQuery file just add: jQuery.noConflict(); and then change all references to the $() to jQuery(). That should do it. Although just using a jQuery modal window script is truly the easiest solution.