Can you be a little more specific? Have you reviewed asterisk and hylafax logs? Yo should see your problems there.
Anyway, if you are going to use a dedicated line for faxing you can consider installing a modem into your server, yes. The performance and reliability of a real modem (or dedicated fax digital card) is much better than using iaxmodems, higher speeds and less problems.
Whatever you do, you will need to learn very well how to use asterisk and hylafax, at least. Read some manuals for that.