What is the meaning of the Custom ISO feature provided by some of the web hosts?
You can upload an ISO of your choice linux distribution to load on your server.
Is it only provided with VPS hosting service?
It would only be provided for VPS or dedicated server offerings.
It's not possible for a shared hosting account to run its own operating system.