300K article, best way to import?

CMS & WordPress
1

Hi,

I have a site made with Yii1.1, the content is over 300K article on dedicated. What would be the best to import it to WP?

THX

2

There’s bound to be a data import plugin for Wordpress somewhere. With that many entries it will probably be a good idea to import in batches to prevent timeouts!

3

Sorry for the delayed reply :wink:
I ended with writing my own shell script to read from old DB and insert and to WP DB, tested and it working fine, but I need to know the best batch size. The sited hosted on dedicated server 16Gb RAM, and SSD.

So, any idea about the patch size?

4

Hard to say, it really boils down to how much data you can import whilst avoiding timeouts. I’d run some tests on some sizable batches, try say 1,000 records first, if that works try higher amounts until it falls over! When you reach a “safe” number that would be the best batch size.

1 Like
5

I will try this, thanks a lot.

6

On the command line you wouldn’t run into time outs but you might run into memory limit issues. Timeout is only a browser/http based setting. Typically that is set to 0 on the command line.

7

I would’ve first tried to save the remote data to CSV file format then import the remote CSV using rsync.

https://rsync.samba.org

Importing data from the old to the new database should be straight forward and perhaps use insert load data because it is exceptionally fast for very large record counts.

https://dev.mysql.com/doc/refman/8.0/en/load-data.html

8

this is will be hard, saving huge data in CSV, moreover, there are too many fields not in Wordpress that require to query it and insert it to WP DB. Also the old site store the big size images in a different directory and store the article image name in the DB, once the article is requested the script will generate a thumbnail of the original and store in the cash … it is not doable with a ready-made plugin.

As of now am facing a new issue, the imported images permission, the image owner and group is the user which ran the script, I can change it from the CLI, but any idea how to fix this from script side?

BTW, I changed the script to PHP that runs from the CLI.

Thanks,

9

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.