I have been running into this same problem. Here are a couple things I have dug up:
In order for the files to store properly in MSSQL, you need to change two lines in file_api.php
Have a look at this post for the exact lines and changes : http://www.mantisbt.org/bugs/view.php?id=6063
Once that is done, your files will be properly stored in the database. However, getting them out is another issue that I am still in the middle of wrestling with. I know the files are now being stored properly, I tested pulling them out with a ColdFusion page and they displayed/saved just fine.
I suspect the issue is somewhere in file_download.php, but I haven't found it yet. Once I do, I'll update this post.