Slot Width First, each string slot cut into the nut should just slightly exceed the diameter of it's string. If the slot is smaller than the string then the string may bind in that slot and create tuning problems. (And that annoying creaking sound when tuning up.)
When changing strings check the slots width by detuning a string, grasping it both before and after the nut and gently pulling it up out of it's slot. If there is any resistance, the slots too small.
An increase in string gauge equals an increase in string diameter and often requires an adjustment to the slots width.
 
Slot Angle Second, the slots must also be angled correctly. The bottom of the nut slot should match the strings natural break angle towards it's tuner. If the angle is too steep the string may actually rest on just a small portion of the slot.  A poorly angled nut slot can create a jangly, buzz sound that can often be silenced if downward pressure is applied to the string behind the nut (over the peghead). This is often a sign that the string is not contacting the nut slot properly.

Slot Depth/ Nut Height And third, once the slots have been deepened you'll want to remove material from the top of the nut, if need be, to avoid having the strings "buried" in the nut. I have improved sustain on many instruments simply by shaving the top of the nut down. Generally speaking I like about half the diameter of the bass strings above the top of the nut, I want the slots deep enough to keep the strings from popping out when plucked but not so deep that they're buried.