Tight clearance bit holder