Flat file databases, heirarchical databases, network databases, and object oriented databases don't even have the concept of relational integrity to enforce. So since referential integrity as a concept isn't even a part of most database model theory there in no way that the implementation of those databases would even attempt to enforce it. Only with a relational database is referential integrity even a concept that the database might choose to enforce and in many cases even relational databases do not properly enforce referential integrity (eg. it is impossible to properly enforce referential integrity where any fields in the database are allowed to exist without having a specific value in the field).
Obviously if you are not going to use the database layer of a particular database model then you'd use a different format for the actual data. Each database uses its own internal file format based on what the actual database processing layer is going to need and so in many cases your generalisation is completely wrong since the data in many database models is designed to be able to be read directly as well as via the database calls. Again you are limiitiing yourself to a specifiic implementation of a database rather than considering databases in general.
A flat file IS a database where you get to build your own access layer on it and so if you built it properly then that database would be the quickest way of accessing your data. So the question of database versus flat file where a flat file can be a database is a rather meaningless question anyway.
You are obviously taking a very narrow view as to just what a database is that only includes one particular database model since relational integrity is not even considered for most types of databases since those models don't require referential integrity as a part of that model.
Providing a standard way of accessing the data that will be the quickest way under normal circumstances is the ONLY thing that all the different types of database have in common.