I’m web-scraping and at the moment I have this code
app.get("/", async (req, res, next) => {
console.log(req.query)
const content = await scrape();
res.json(content);
});
Using puppeteer I’m scraping a website with has a lot of images and roughly the end resulr is this:
const scrape = async () => {
const url = "http://...";
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url);
....
return {
artistPhotos,
topAlbums,
topTracks,
artistName: document.querySelector(".header-new-title").innerText,
styling,
stats,
latestRelease,
popularThisWeek,
description,
similarArtists,
};
}
In the front end, I have to wait for a long time till all that content is scraped. I need all that content in to show on the page.
(function(){
fetch(url)
.then(res => res.json())
.then(data => {
console.log(data)
domManipulation(data)
} )
})()
I could have a loading wheel, but what is the best way to lead that data in chucks?
I was thinking
(function(){
fetch('http://api?type=artistPhoto')
.then(res => res.json())
.then(data => {
console.log(data)
domManipulation(data)
fetch('http://api?type=topAlbums')...
...
} )
})()
Is this the right way to do it or is there a better way