Technically, you can achieve your goal without an open-source licence. There's nothing to stop you form selling the software on a commercial basis, retaining the copyright, but giving access to the source code with permission to modify it.
But the obvious problem is that there will be no way of policing it. If you expect your users to pay you a commission for each book sold, it will be easy for the less scrupulous among them to modify the code so that it no longer reports the sales or makes the payment.
So, from a technical and legal point of view, what you want to achieve is perfectly possible, but from a business point of view, it is highly risky. Maybe you should look for some other business model for exploiting this market.