Nearly 20 years ago we started analysing medical transactions. Typically the data would be in a delimited txt format, collected from a number of sources. Initially we used SQL but after running into a number of issues we built our own “engine” (Win Api, TopSpeed C & Clarion). Database sizes have ranged from tens of millions to billions of records. The outputs are mostly in an agreed standard format but also include ad hoc analysis.
If you’re having difficulty analysing zillions of records quickly and cheaply, consider outsourcing the function.