I Don't Know Who Got Me Pregnant: My Husband Or My Boyfriend?

[img http://3.bp.blogspot.com/-LCgyyfch69I/UFRYKjZp-DI/AAAAAAAANvI/kH1WxU0OXDU/s1600/WOMAN PREGNANT.jpg[/img] People should not judge me but help me understand what I've gotten myself into and how to come out of these complications. Pls I can't disclose my name for obvious reasons.
Let me just go straight to the point. I'm a good wife to my husband and he loves me too. We are both sex freaks, sometimes we even make love in the kitchen while am cooking since we live alone. That's how sweet our marriage was. But not any more.
Over four years into our union no"evidence" and pressures from my family and his were getting unbearable so I took to a colleague's advice. I started dating a young guy but this guy always insist on using condom onme, claiming he can only have direct sex with a woman that is ready to get pregnant for him.
I told him I was ready and we started doing skin-to-skin for about three (3) months before myhusband finally called me one weekend. Meanwhile, I didn't know my colleague has told this my boyfriend some things about my personal life, especially my husband's "inability" to impregnate me.
Back to my husband: he called meand said he was ready for us to go see a doctor so we could know what is wrong as I had earlier suggested. I was happy because it would help us get medical help if something was actually wrong with either of us.
But I got much more than I bargained for when the doctor came back with the results of the tests that were carried out on us. He said my husband and I are perfectly ok and even congratulated my husband. Congratulation for what? I asked the doctor.
He said my husband has done it, I was already six weeks pregnant! Iwas filled with joy and my husband too. We celebrated the good news with several rounds of passionate sex when we got home that day. I'm now five months heavy.
I've stopped seeing my boyfriend and also resigned from work. Somehow my colleague told this guy I was pregnant and he came looking for me in our house. He said I ran away with his pregnancy, bla bla bla, saying I should do the honourable thing and give him his baby after the baby is delivered. Since that day, I've not been myself.
He now calls me almost every afternoon to ask how my baby is doing and other rubbish. My husband is beginning to suspect that something is bothering me but I've been taking care of his side. However, my BIG problem is this guy. I don't think he's the owner of my pregnancy.
After all, the doctor said nothing is wrong with my husband and I believe my husband actually got me pregnant but I can't tell him about this guy, it will destroy our marriage.
I need people's advise on how to handle this guy once and for all. Please help me!!!!