How to send json data in chuncks using express.js?

JavaScript
1

I’m web-scraping and at the moment I have this code

  app.get("/", async (req, res, next) => {
    console.log(req.query)
    const content = await scrape();
    res.json(content);
  });

Using puppeteer I’m scraping a website with has a lot of images and roughly the end resulr is this:

const scrape = async () =>  {
const url = "http://...";
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(url);
....
return {
        artistPhotos,
        topAlbums,
        topTracks,
        artistName: document.querySelector(".header-new-title").innerText,
        styling,
        stats,
        latestRelease,
        popularThisWeek,
        description,

        similarArtists,
      };
}

In the front end, I have to wait for a long time till all that content is scraped. I need all that content in to show on the page.

(function(){
fetch(url)
    .then(res => res.json())
    .then(data =>   {
        console.log(data)
        domManipulation(data)
    } )
})()

I could have a loading wheel, but what is the best way to lead that data in chucks?

I was thinking

(function(){
fetch('http://api?type=artistPhoto')
    .then(res => res.json())
    .then(data =>   {
        console.log(data)
        domManipulation(data)
       fetch('http://api?type=topAlbums')...
...
    } )
})()

Is this the right way to do it or is there a better way

2

Is all of this data coming from one page or multiple pages?

In the case of multiple pages, this might help:

2 Likes
3

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.