I had the same issue and have done exactly what you have done with filing taxes yearly and not doing the Form 1040 based on the Schedule C as intended. My accountant says that it's frowned upon because if you do owe taxes, you can end up having to pay interest on owed taxes for the year. It's the same interest as if you're late on paying the full amount on April 15th.

So, no, the IRS won't come after you, but yes, you might end up paying more than you're supposed to if you do not file the estimated Form 1040-ES quarterly. Also, you'll only get the 1099's once, at the beginning of the completed year, so your payments are not based on the 1099's because typically you haven't received it yet.