Your baby has no control of what he is doing and cannot by reasoned with, any threats of anger or outbursts are useless and terribly damaging. Physical aggression or force is never okay and can lead to broken bones, brain damage, or worse. It's not malicious, or spiteful, or mean. It's just how they are, and it's all they can do to communicate.
Your child needs you to be his protector, not predator. Parental aggression (especially from the father) is a real phenomena, especially with colic babies.
Recognize when your frustration is rising and quickly remove yourself from your baby and wife (assuming they are in a safe place together). Sleep in the living room, or even out in your car to separate the possibility of physical contact between you and your child.
Return only when your temper is back down.
Talk to your wife so she understands what's happening at moments when you suddenly need some space. Trying to explain in the moment could result in you just getting more frustrated if she's not understanding quickly enough.
Breathe deep, take a break, step away. Parenting is hard, and if you act out when your child needs you it can be even harder.