we’re currently using an open source CRM system for small company and have a custom WordPress plugin that integrates with our CRM via REST.
Our clients can login into the WP plugin and see their data as we store it in the CRM. (like cases, projects, contracts etc.)
Our biggest issues are
SSO - the auth against the two systems with all possible scenarios (forgot password, change password, different roles, etc)
Speed - portal and CRM are on two different servers + the CRM comes with all the CRM overhead in the API and SQL queries
Flexibility - new ideas can’t be configured (like navigation menus in WP) but often need to be custom developed
A few years ago, I’ve worked on many Drupal installations for a decade and I was wondering (would likely work with WordPress as well):
If we were to create custom content types/post types and configure their views in Drupal/Wordpress and “simply” develop a sync software that does its bi-directional sync, we don’t have to worry about slow REST or SSO anymore.
On another project, I’ve created an unidirectional sync from a shipping software into a CRM and that was working much better than the current project. But it was much smaller as well.
My question now is: Does anybody have experience with a similar move - from API integration to a background sync for a business portal type of web application?
Our next big step on the roadmap is towards a SaaS version, and I’d be happy to lay the right foundation for this one already.
Bi-directional sync is hard, because you loose the book of record. If one system says the state of something is A and the other one says B, which one is right?
If I were in your situation I would focus more on improving the current situation by using smarter calls, caching stuff, that sort of thing.
Caching seems to be a challenge. Possible definitely, but it seems we’d need to create something like a script that uses all users to login into the portal and cache their records automatically (if we want to provide even a fast first page load to the client).
What about the security and data aspect?
We’d like to put our CRM into a VPN or similar, since currently the CRM and it’s REST API are exposed to the internet.
A bi-directional sync would help there.
(We are very slow with upgrades due to internal priorities / understaffed … still on PHP5.6 and passed the EOL of our LTS CRM since a couple of month - this situation is bad, but we cannot upgrade easily with a ton of customizations and our CEOs attitude for features)
It is fairly common to create convoluted, expensive work arounds to improve the inefficiencies of a third party product that an organization has committed itself to. What you are proposing sounds like a reasonable solution to me when considered in this context.
You will most definitely want to run this process on a separate server from your app server. You will also want to run it on the command line or use continuous integration software like Jenkins to create jobs, run, and monitor their health. I would probably opt out of using a relational database for this. Using a NoSQL database would be less overhead and hassle. Just fetch objects from the api and place them in mongo or something and not need to worry about setting up a bunch of tables and relationships that could potentially break. A NoSQL database would be extremely fast even if you where to wrap it in an api and call that from wp. Of course you are now creating a completely new software product that will need to be maintained and monitored. Not to mention introducing new technology which people will need learn to maintain this properly separate from wp which is a whole other skill set really. I would also probably than suggest rebuilding the ui in a modern js framework like react or vue using web components and have a wp plugin that just bundles and puts that all on the page instead of dealing with integrating directly into wp which is probably an exercise in futility. If you do all that I’m sure the user experience would be blazing fast.
If the data source is slow, it doesn’t matter how fast the layer is that you stack on top of it. Worst case scenario it’s still slow. Average case may drop, sure.
That isn’t true if you set-up jobs to sync data into mongo. The web application would fetch data from mongo instead of the original api. Which would make it much faster.
How would you know if the data in mongo is still fresh? What happens when the CRM is updated and the website is not, and someone on the website issues a command that is not possible in the CRM due to the updates that happened there?
As a general rule I tend to fight the root cause of problems instead of battling symptoms. So in this case my preferred solution would be to update the entire stack first, see what happens then in terms of performance, and if its not enough find the use cases that are slow and find an easy solution to optimize those.
It’s so easy to add more complexity. That doesn’t necessarily solve your problem though. More often than not it makes it worse.
The ideal solution would be web hook to update the data but I doubt that is available. Therefore, it would need to be done on some type of schedule with an understanding that the data will not always be fresh.
So the OP would go from slow but up to date to faster but not sure if up to date. And time (and thus money) needs to be invested to get there. At some point money and time would also need to be spent to upgrade all systems, regardless of whether this intermediary solution is implemented or not. When it turns out the problems go away with the update, the intermediary solution can (and should!) be thrown out.
That would be a decision for the business to make.
I’m not certain, whether I’m clear on the difference.
If we were to integrate with WP, we’d be using WP API / database tables etc. to manage users, content etc.
If we were to build a plugin, we’d build our app as a WP plugin and the only required integration would for instance be the user management?
Jenkins? I think we’re arriving at a level of complexity that our small company is currently not able to handle.
Is, in your opinion, the approach already wrong?
If we see the CRM as our marketing/sales software and the webapp as theWP/ webapp, what shall we do with the overlapping part, where we have busines processes in the CRM that we’d like to offer now to our clients as a SaaS?
The approach by itself Im not sure, but this is absolutely wrong:
We are very slow with upgrades due to internal priorities / understaffed … still on PHP5.6 and passed the EOL of our LTS CRM since a couple of month - this situation is bad, but we cannot upgrade easily with a ton of customizations and our CEOs attitude for features
PHP 7 is loads faster than PHP 5.6. The CRM likely has speed improvements as well. I’d start there. See what the performance is like when everything is up to date.
Also try the move the servers closer together. I don’t know how far apart they are now, but the closer they are together the better.
Always start with the simple stuff (well in this case the upgrades may not be simple, but they need to be done anyway, so…). Adding more complexity can always still be done when the simple stuff doesn’t work.
In terms of SSO, how is that set up? Is the CRM the SSO provider and the website the SSO consumer?
Yes, I agree about the outdated stuff. Security holes presented to the web and everything is slow and old. I try to convince the CEO to allocate more time for maintenance, but that’s something new for him as it seems.
SSO is done from a WordPress plugin after the WP login, users authenticate against our CRM customers. But we’ve built in exceptions for some WP plugins which handle their own users (affiliate, admin, …) and we didn’t get all parts working of SSO (forgot password, …)
A pretty patchy solution in my opinion but again: We’ve been working with and on this plugin for 3 years with the same developer company - there is no option for a quick change.
Yes, at times, it really feels like this would be the right solution 