Defamatory?

I have recently been swindled out of thousands of pounds of money by a business man. I deciced to create a web site in his name publishing all of the wrong doings he committed to me and to others. The man is claiming defamatory and is getting the ISPs ( Internet Service Providers) to remove the content. Knowing that if all the facts I have wrote about this man are true, should I be worried about his threats of legal action.