Comparing data

(3) [Array(2), Array(2), Array(2)]
0: (2) ["Twitter", "https://www.asdf.com/fdsa/320738?filter=605975,635426"]
1: (2) ["Instagram", "https://www.asdf.com/fdsa/320738?filter=649104"]
2: (2) ["Facebook", "https://www.asdf.com/fdsa/320738?filter=616690"]

I have this multi dimensional array.I need to loop over each array item here.

(3) ["instagram&per=1&page=1", "twitter&per=2&page=1", "facebook&per=1&page=1"]
0: "instagram&per=1&page=1"
1: "twitter&per=2&page=1"
2: "facebook&per=1&page=1"

I also have this array.

While looping over the first array, I need to ā€œfindā€ the match in the 2nd array, and take the first arrays URL, and append the rest of the 2nd array.

So e.g. the first loop will go over this.

0: (2) ["Twitter", "https://www.asdf.com/fdsa/320738?filter=605975,635426"]

It should match:

1: "twitter&per=2&page=1"

The final value will be pushed into a new array:

https://www.asdf.com/fdsa/320738?filter=605975,635426&per=2&page=1

I’m not sure of how I should approach this in code, unfortunately.

Pre-process the second array and create another array, something like:

var array3= [];
array3["instagram"] = "per=1&page=1";
array3["twitter"] = "per=2&page=1";
array3["facebook"] = "per=1&page=1";

You can use the indexOf() Method to find the first ā€œ&ā€ in each element of the second array. I am not sure if you want to retain the first ā€œ&ā€ in the values. And the ā€œinstagramā€, ā€œtwitterā€, ā€œfacebookā€ and other values would not be hard-coded in the source code, they would be obtained when you process the second array.

Then when you process the first array you can easily look up the relevant entry in the third array, except be sure to ensure that the case is not a problem. In other words ā€œTwitterā€ will not match ā€œtwitterā€.

Well it’s inefficient, but…
First i would smash your second array into an object, with the names as keys.

let obj = Object.fromEntries(array2.map((x) => { let y = x.split("&"); return [y.shift().toLowerCase(),"&"+y.join("&")] }))

then walk your first array and merge as necessary:
array1 = array1.map((x) => x[1] = x[1]+(obj[x[0].toLowerCase()] || "")

But i feel there’s probably a cleaner way.

1 Like

Thanks to your ideas, I was able to get this. This is the completed function

combineURLData(filteredURLs, options) {
    var finalURLs = [];
    var urlData = Object.entries(filteredURLs);
    urlData.forEach(function(data, index) {
      for(let i=0;i<urlData.length;i++) {
        if(data[0].toLowerCase() === options[i].split("&")[0]) {
          finalURLs.push(data[1]+"&"+options[i].substring(options[i].indexOf("&")+1));
        }
      };
    });
    return finalURLs;
  }

Where filteredURLs is the first array, and options is the second array in my OP.

…But I have a new problem.
So, this is what initiates everything
this.urls = this.createURLsFromFilters(endpoint);
It’s called in function A.

createURLsFromFilters(endpoint) {
    var filters = Object.entries(this.filters);
    var urls = Object.entries(endpoint.split("?filter=").pop().split(","));
    var urls2 = urls.map(filter => endpoint.split("?")[0] + "?filter=" + filter[1]);
    var sourceData = {};
    Promise.all(urls2.map(url =>
      fetch(url).then(resp => resp.json())
    )).then(values => {
      values.forEach(function(value, index) {
        if(value.posts.items[0].source.source in sourceData) {
          sourceData[value.posts.items[0].source.source] = sourceData[value.posts.items[0].source.source] + "," + urls2[index].split("?filter=")[1];
        } else {
          sourceData[value.posts.items[0].source.source] = endpoint.split("?filter")[0] + "?filter=" + urls2[index].split("?filter=")[1];
        }
      });
      var filterOptions = filters.map(filter => filter[0] + "&per=" + filter[1] + "&page=" + this.options.feed.page);
      var finalData = this.combineURLData(sourceData, filterOptions);

      return finalData;
    });
    // Old inaccurate data
    // return filters.map(filter => endpoint.split("?")[0] + "?filter=" + filter[0] + "&per=" + filter[1] + "&page=" + this.options.feed.page);
  }

That function calls a separate function to do more stuff

combineURLData(filteredURLs, options) {
    var finalURLs = [];
    var urlData = Object.entries(filteredURLs);
    urlData.forEach(function(data, index) {
      for(let i=0;i<urlData.length;i++) {
        if(data[0].toLowerCase() === options[i].split("&")[0]) {
          finalURLs.push(data[1]+"&"+options[i].substring(options[i].indexOf("&")+1));
        }
      };
    });
    return finalURLs;
  }

This function right here is what gives me my final (and correct) URLs. It’s returned to createURLsFromFilters. I need to return THAT to function A, but my return finalURLs doesn’t make it back. It’s returning undefined.

@m_hutley

A bit of trial and error with split ?? A splitOnce would be nice.

const socialUrlProps = [
  'instagram&per=1&page=1', 
  'twitter&per=2&page=1', 
  'facebook&per=1&page=1'
]

const mapped = Object.fromEntries(
  socialUrlProps.map(prop => prop.split(/(&[^]+)/, 2))
)

console.log(mapped)
/*
{
  instagram: "&per=1&page=1", 
  twitter: "&per=2&page=1", 
  facebook: "&per=1&page=1"
}
*/
1 Like

Anyone have any idea how to get my return to actually return my data?

Yeah, split’s limit is… a bit annoying, with the dropping of excess text. I honestly didn’t think about using a greedy regex. You could have simplified it to .+ instead of [^]+, since we don’t need to deal with line breaks, but good shout.

What’s your function not returning that you’d want it to? Maybe I missed something…

Let me try explaining more about how this program runs.
I have a function that sets this:
this.urls = this.createURLsFromFilters(endpoint);
Right after this line of code is a Promise/then (much like in my last post). It goes over this.urls.

this.urls is the unformatted URL that I need to do a lot of work on.

The problem is that the Promise is not waiting to run before this.urls is completed. So it’s an empty / undefined variable (the program isn’t up right now but on Monday I can give actual console logs)

So, createURLsFromFilters is called (passing the unformatted URL) and from that function, I have a Promise in that (see last post). This Promise basically splits the unformatted URL into a separate URL per numbered query string.

So e.g.
https://www.asdf.com/fdsa/320738?filter=605975,635426

Gets turned into

https://www.asdf.com/fdsa/320738?filter=605975
https://www.asdf.com/fdsa/320738?filter=635426

However many numbers are there. This is because I need these individualized URLs like this so I can parse the URL for specific JSON data.

So, if you are still following, we are in the then part of createURLsFromFilters. We have the individualized URLs as listed above. I then take each of those URLs (example above), and this secondary array…

(3) ["instagram&per=1&page=1", "twitter&per=2&page=1", "facebook&per=1&page=1"]
0: "instagram&per=1&page=1"
1: "twitter&per=2&page=1"
2: "facebook&per=1&page=1"

And I use these arrays to call combineURLData. This gives me my final URLs.

0: "https://www.asdf.com/fdsa/320738?filter=649104&per=1&page=1"
1: "https://www.asdf.com/fdsa/320738?filter=605975,635426&per=2&page=1"
2: "https://www.asdf.com/fdsa/320738?filter=616690&per=1&page=1"

So you can see, it takes a while for this.urls to complete. But the Promise (not shown here) that gets called immediately after this.urls doesn’t wait for that to complete, so it’s empty.

Hopefully with that context, you can re-read my last post and it makes sense? Otherwise, I’ll provide more insight on Monday to whatever is confusing you still.

Here we go. So this is the function that starts it all:

requestPostData(endpoint, feedOptions, callback) {
    this.urls = this.createURLsFromFilters(endpoint);
    console.log("this.urls");
    console.log(this.urls);
    Promise.all(this.urls.map(url =>
      fetch(url).then(resp => resp.json())
    )).then(values => {
      callback(values);
    });
  }

Note: in all the console logs in my program, this one gets outputted first, and I get an undefined error

this.urls
undefined
Uncaught TypeError: Cannot read property 'map' of undefined

Anyway, that this.urls line calls createURLsFromFilters. I’ll post the relevant code from that function.

var finalData = this.combineURLData(sourceData, filterOptions);
      console.log("Final Data");
      console.log(finalData);
      return finalData;

This ā€œFinal Dataā€ console log is actually my last console log in my program. My console log in combineURLData (below) gets console.log next. Both of my final console logs are the same data, but I’m just illustrating the order.

So this is the combineURLData function

combineURLData(filteredURLs, options) {
    var finalURLs = [];
    var urlData = Object.entries(filteredURLs);
    urlData.forEach(function(data, index) {
      for(let i=0;i<urlData.length;i++) {
        if(data[0].toLowerCase() === options[i].split("&")[0]) {
          finalURLs.push(data[1]+"&"+options[i].substring(options[i].indexOf("&")+1));
        }
      };
    });
    console.log("combineURLData finalURLs");
    console.log(finalURLs);
    return finalURLs;
  }

This is the 2nd console log group. It’s returned to createURLsFromFilters (my finalData snippet). Then that finalData is returned to my intial function call.

That should clear up any confusion, I hope.

I’m relatively new to Promises, but have to say not a fan of the above. I have just seen similar solutions posted on stackoverflow, and it seems to defeat the purpose of promises whereby you avoid nested callbacks.

Just experimenting with an alternative and the following seems to work

const urls = [
  'https://jsonplaceholder.typicode.com/todos/1',
  'https://jsonplaceholder.typicode.com/todos/2',
  'https://jsonplaceholder.typicode.com/todos/3'
]

// I don't want map's index being passed as fetch's optional credentials argument
// so will lock it to one argument
const fetchUrl = (url) => fetch(url)
const getJson = response => response.json()

Promise.all(urls.map(fetchUrl))
  .then(allResponses => Promise.all(allResponses.map(getJson)))
  .then(console.log)

Output

// output
0: {userId: 1, id: 1, title: "delectus aut autem", completed: false}
1: {userId: 1, id: 2, title: "quis ut nam facilis et officia qui", completed: false}
2: {userId: 1, id: 3, title: "fugiat veniam minus", completed: false}

It reads a little more clearly to me

this.createURLsFromFilters(endpoint) is an asynchronous operation, so console.log("this.urls", this.urls) which is part of the global execution context will be executed long before createURLsFromFilters has done it’s work successfully

this part

Promise.all(this.urls.map(url =>
      fetch(url).then(resp => resp.json())
    )).then(values => {
      callback(values);
    });

needs to be part of the chain of events inside createURLsFromFilters e.g. a callback to be called when createURLsFromFilters has successfully got the required data

edit: I think

This is very rough around the edges and obviously I can’t test it, but I think is going in the right direction

// helpers
const fetchUrl = (url) => fetch(url)
const getJson = response => response.json()

createURLsFromFilters(endpoint, callback) {
  const filters = Object.entries(this.filters);
  const urls = Object.entries(endpoint.split("?filter=").pop().split(","));
  const urls2 = urls.map(filter => endpoint.split("?")[0] + "?filter=" + filter[1]);
  const sourceData = {};

  Promise.all(urls2.map(fetchUrl))
    .then(allResponses => Promise.all(allResponses.map(getJson)))
    // this could do with refactoring
    .then(values => {
      
      // this callback inside forEach might be better written as a separate function
      values.forEach(function(value, index) {
        if(value.posts.items[0].source.source in sourceData) {
          sourceData[value.posts.items[0].source.source] = sourceData[value.posts.items[0].source.source] + "," + urls2[index].split("?filter=")[1]
        } else {
          sourceData[value.posts.items[0].source.source] = endpoint.split("?filter")[0] + "?filter=" + urls2[index].split("?filter=")[1]
        }
      })

      // maybe filterOptions could be a function that returns this data
      const filterOptions = filters.map(filter => filter[0] + "&per=" + filter[1] + "&page=" + this.options.feed.page)

      // urls passed to createURLsFromFilters' callback
      callback(this.combineURLData(sourceData, filterOptions))
    })
}

requestPostData(endpoint, callback) {
  this.createURLsFromFilters(
    endpoint, 
    // callback to pass to createURLsFromFilters
    (urls) => {
      Promise.all(urls.map(url =>
        fetch(url).then(resp => resp.json())
      )).then(values => {
        callback(values);
      })
    }
  )
}

Edit: A bit of refactoring. Note all theory as I haven’t been able to test it

// helpers
const fetchUrl = (url) => fetch(url)
const getJson = (response) => response.json()
const getFirstProp = (strg, term) => strg.split(term)[0]
const getLastProp = (strg, term) => strg.split(term).pop()

// extracted out into own function
const getSourceData = (values, urls) => (
  values.reduce((data, value, index) => {
    const postItemSource = value.posts.items[0].source.source

    data[postItemSource] = (postItemSource in data)
      ? `${data[postItemSource]},${getLastProp(urls[index], '?filter=')}`
      : `${getFirstProp(endpoint, '?filter')}?filter=${getLastProp(urls[index], '?filter=')}`

    return data
  }, {} // <-- this is the original data object)
)

// method of an object I presume?
const createURLsFromFilters = function(endpoint, callback) {
  const filters = Object.entries(this.filters);

  // note: name1, name2 should be replaced with more descriptive names
  const urls = 
    Object
      .entries(getLastProp(endpoint, '?filter=')
      .split(','))
      .map(([_, name2]) => `${getFirstProp(endpoint, '?')}?filter=${name2}`)

  Promise.all(urls.map(fetchUrl))
    .then(allResponses => Promise.all(allResponses.map(getJson)))
    .then(values => {
      const filterOptions = filters.map(([name1, name2]) => `${name1}&per=${name2}&page=${this.options.feed.page}`)

      callback(this.combineURLData(getSourceData(values, urls), filterOptions))
    })
}

requestPostData(endpoint, callback) {
  this.createURLsFromFilters(
    endpoint,
    // callback to pass to createURLsFromFilters
    urls => {
      Promise.all(this.urls.map(url =>
        fetch(url).then(resp => resp.json())
      )).then(values => {
        callback(values);
      })
    }
  )
}

note I am making use of destructing
array.map(([x, y]) => [x, y]) is the same as array.map((values) => [values[0], values[1]])

edit: I will try and do a better breakdown of the above

A bit more refactoring. The following has worked in simple tests, but may need to be a bit more bulletproof.

I was thinking getJson could be re-written to handle promiseAll and individual promises, just by checking if the returned value is an Array.

const getJson = (response) => (
  (Array.isArray(response))
    ? Promise.all(response.map(getJson))
    : response.json()
)

so for instance the following

requestPostData(endpoint, callback) {
  this.createURLsFromFilters(
    endpoint,
    // callback to pass to createURLsFromFilters
    urls => {
      Promise.all(this.urls.map(url =>
        fetch(url).then(resp => resp.json())
      )).then(values => {
        callback(values);
      })
    }
  )
}

can then be refactored to

requestPostData(endpoint, callback) {
  this.createURLsFromFilters(
    endpoint,
    (urls) => {
      Promise.all(urls.map(fetchUrl))
        .then(getJson)
        .then(callback)
    }
  )
}

Not sure if I’m helping or hindering. Admittedly have got a bit carried away.

// helpers
const fetchUrl = (url) => fetch(url)
const getFirstSplit = (term, strg) => strg.split(term)[0]
const getLastSplit = (term, strg) => strg.split(term).pop()

const getJson = (response) => (
  (Array.isArray(response))
    ? Promise.all(response.map(getJson))
    : response.json()
)

const getSourceData = (values, urls, endpoint) => (
  values.reduce((data, value, i) => {
    const postItemSource = value.posts.items[0].source.source
    const filterProps = getLastSplit('?filter=', urls[i])

    data[postItemSource] = (postItemSource in data)
      ? `${data[postItemSource]},${filterProps}`
      : `${endpoint}?filter=${filterProps}`

    return data
  }, {})
)

const getUrls = (endpointPath, filter) => (
  Object
    .entries(filter.split(','))
    .map(([_, filter]) => `${endpointPath}?filter=${filter}`)
)

const getFilters = (filters, page) => (
  Object
    .entries(filters)
    .map(([filter1, filter2]) => `${filter1}&per=${filter2}&page=${page}`)
)

createURLsFromFilters (endpoint, callback) {

  const urls = getUrls(
    getFirstSplit('?', endpoint), // path
    getLastSplit('?filter=', endpoint) // filter
  )

  const filters = getFilters(this.filters, this.options.feed.page)
  const sourceData = getSourceData(values, urls, endpoint)
  const getPostData = values => callback(this.combineURLData(sourceData, filters))

  Promise.all(urls.map(fetchUrl))
    .then(getJson)
    .then(getPostData)
}

const requestPostData (endpoint, callback) {
  this.createURLsFromFilters(
    endpoint,
    (urls) => {
      Promise.all(urls.map(fetchUrl))
        .then(getJson)
        .then(callback)
    }
  )
}

Just a bit of a side note, I’m sure this could be improved on, but behind my thinking here…

A few shorter functions is preferable to one large more complicated function. Easier to read, less to go wrong and easier to test.

Thank you! I’ll need some time to process all this and see how I can integrate it. I’ll get back to you shortly.

1 Like

I’ve been trying to wrap my head around all your code, but I’m a bit lost. Can you break it down a bit, please? Here is the whole code:

// ====================================================
// Custom Finalsite Feeds Plugin
// ====================================================

import {
  nano
} from './utilities';

class FSFeeds {
  constructor(target, options, callback) {
    this.target = target;
    this.endpoint = target.attr('data-feed-url');
    this.options = options;
    this.postData;
    this.defaultTemplate = ['<article class="fsFeeds-post">',
        '<div class="post-image" style="background-image: url({image})">',
          '<img src="{image}" class="post-image-inline" />',
        '</div>',
        '<div class="post-content">',
          '<div class="like-count">{like_count}</div>',
          '<div class="post-description">{date} {unformatted_message}</div>',
        '</div>',
      '</article>'];
    this.dateFormat = options.dateFormat || "MMMM Do YYYY";
    this.templates = options.templates || { 'Facebook': this.defaultTemplate, 'Twitter': this.defaultTemplate, 'Instagram': this.defaultTemplate, 'YouTube': this.defaultTemplate };
    this.loadMoreTemplate = options.loadMoreTemplate || "<button>Load More</button>";
    this.renderedTemplate = [];
    this.callback = callback;
    this.loadMore = options.loadMore;
    this.filters = options.filters || {};
  }

  requestPostData(endpoint, feedOptions, callback) {
    $.getJSON(endpoint, feedOptions, function(data) {
      }.bind(this)).done(function(data) {
        callback(data.posts.items);
      }.bind(this)).fail(function() {
        this.target.append('<span>Sorry, an error occured when retrieving this feed data. Please refresh the page to try again.</span>').css('textAlign', 'center');
    });
  }

  getTemplate(type) {
    return this.templates ? this.templates[type] || this.defaultTemplate : this.defaultTemplate;
  }

  construct() {
    var template = []
    this.postData.forEach((post) => {
      var post = post;
      var postType = post.source.source;
      if(postType == "Instagram" && post.additional_photos && post.additional_photos.length) {
        post.image = post.additional_photos[0]
      }
      post.date = moment(post.external_created_at).format(this.dateFormat);
      template.push(nano(this.getTemplate(postType).join('\n'), post))
    });
    return template;
  }

  render(template) {
    this.target.find("> .fsElementContent").append(template.join(''));
  }

  renderLoadMore() {
    this.target.append($(this.loadMoreTemplate).addClass("fsFeeds-load-more"));
    $(document).on("click", ".fsFeeds-load-more", () => {
      this.options.feed.page = this.options.feed.page + 1;
      this.requestPostData(this.endpoint, this.options.feed, (data) => {
        this.postData = this.postOrder ? this.sortPosts(data) : data;
        this.renderedTemplate = this.construct(this.postData);
        this.render(this.renderedTemplate);
        this.callback();
      });
    });
  }

  sortPosts(posts) {
    var sorted_posts = [];
    this.postOrder.forEach( function(feed) {
      var postToSort = posts.find(function(post) {
        return post.source.source == feed;
      });
      sorted_posts.push(postToSort);
      posts = posts.filter(function(post) {
        return post !== postToSort
      });
    });
    return sorted_posts;
  }

  init() {
    this.options.feed.page = this.options.feed.page || 1;
    if(this.loadMore) {
      this.renderLoadMore();
    }
    this.requestPostData(this.endpoint, this.options.feed, (data) => {
      this.postData = this.postOrder ? this.sortPosts(data) : data;
      this.renderedTemplate = this.construct(this.postData);
      this.render(this.renderedTemplate);
      this.callback();
    });
  }
}

class MultiRequestFeed extends FSFeeds {

  combineURLData(filteredURLs, options) {
    var finalURLs = [];
    var urlData = Object.entries(filteredURLs);
    urlData.forEach(function(data, index) {
      for(let i=0;i<urlData.length;i++) {
        if(data[0].toLowerCase() === options[i].split("&")[0]) {
          finalURLs.push(data[1]+"&"+options[i].substring(options[i].indexOf("&")+1));
        }
      };
    });
    console.log("combineURLData finalURLs");
    console.log(finalURLs);
    return finalURLs;
  }

  createURLsFromFilters(endpoint) {
    var filters = Object.entries(this.filters);
    var urls = Object.entries(endpoint.split("?filter=").pop().split(","));
    var urls2 = urls.map(filter => endpoint.split("?")[0] + "?filter=" + filter[1]);
    var sourceData = {};
    Promise.all(urls2.map(url =>
      fetch(url).then(resp => resp.json())
    )).then(values => {
      values.forEach(function(value, index) {
        if(value.posts.items[0].source.source in sourceData) {
          sourceData[value.posts.items[0].source.source] = sourceData[value.posts.items[0].source.source] + "," + urls2[index].split("?filter=")[1];
        } else {
          sourceData[value.posts.items[0].source.source] = endpoint.split("?filter")[0] + "?filter=" + urls2[index].split("?filter=")[1];
        }
      });
      var filterOptions = filters.map(filter => filter[0] + "&per=" + filter[1] + "&page=" + this.options.feed.page);
      var finalData = this.combineURLData(sourceData, filterOptions);
      console.log("Final Data");
      console.log(finalData);
      return finalData;
    });
    // Old inaccurate data
    // return filters.map(filter => endpoint.split("?")[0] + "?filter=" + filter[0] + "&per=" + filter[1] + "&page=" + this.options.feed.page);
  }

  requestPostData(endpoint, feedOptions, callback) {
    this.urls = this.createURLsFromFilters(endpoint);
    console.log("this.urls");
    console.log(this.urls);
    Promise.all(this.urls.map(url =>
      fetch(url).then(resp => resp.json())
    )).then(values => {
      callback(values);
    });
  }

  renderLoadMore() {
    this.target.append($(this.loadMoreTemplate).addClass("fsFeeds-load-more"));
    $(document).on("click", ".fsFeeds-load-more", () => {
      this.options.feed.page = this.options.feed.page + 1;
      this.requestPostData(this.endpoint, this.options.feed, (data) => {
        this.postData = this.combinePosts(data);
        this.renderedTemplate = this.construct(this.postData);
        this.render(this.renderedTemplate);
        this.callback();
      });
    });
  }

  combinePosts(data) {
    return data.map(source => source.posts.items).reduce((acc, val) => acc.concat(val), []);
  }

  init() {
    this.options.feed.page = this.options.feed.page || 1;
    if(this.loadMore) {
      this.renderLoadMore();
    }
    this.requestPostData(this.endpoint, this.options.feed, (data) => {
      this.postData = this.combinePosts(data);
      this.renderedTemplate = this.construct(this.postData);
      this.render(this.renderedTemplate);
      this.callback();
    });
  }
}

$.fn.fsFeedPull = function( options, callback ) { new FSFeeds( this, options, callback ).init(); }
$.fn.fsMultiFeedPull = function( options, callback ) { new MultiRequestFeed( this, options, callback ).init(); }
export {
  FSFeeds,
  MultiRequestFeed
};

I think I got it working by combining the 3 functions into these 2, but this probably could use some serious refactoring…

combineURLData(filteredURLs, options) {
    var finalURLs = [];
    var urlData = Object.entries(filteredURLs);
    urlData.forEach(function(data, index) {
      for(let i=0;i<urlData.length;i++) {
        if(data[0].toLowerCase() === options[i].split("&")[0]) {
          finalURLs.push(data[1]+"&"+options[i].substring(options[i].indexOf("&")+1));
        }
      };
    });
    return finalURLs;
  }

  requestPostData(endpoint, feedOptions, callback) {
    var filters = Object.entries(this.filters);
    var urls = Object.entries(endpoint.split("?filter=").pop().split(","));
    var urls2 = urls.map(filter => endpoint.split("?")[0] + "?filter=" + filter[1]);
    var sourceData = {};
    Promise.all(urls2.map(url =>
      fetch(url).then(resp => resp.json())
    )).then(values => {
      values.forEach(function(value, index) {
        if(value.posts.items[0].source.source in sourceData) {
          sourceData[value.posts.items[0].source.source] = sourceData[value.posts.items[0].source.source] + "," + urls2[index].split("?filter=")[1];
        } else {
          sourceData[value.posts.items[0].source.source] = endpoint.split("?filter")[0] + "?filter=" + urls2[index].split("?filter=")[1];
        }
      });
      var filterOptions = filters.map(filter => filter[0] + "&per=" + filter[1] + "&page=" + this.options.feed.page);
      var finalData = this.combineURLData(sourceData, filterOptions);
      return finalData;
    }).then(function(data) {
      console.log(data);
      Promise.all(data.map(url =>
        fetch(url).then(resp => resp.json())
      )).then(values => {
        callback(values);
      });
    });
  }

One small bit of refactoring, and this seems to be quite common place. You are wrapping your call back in an unnecessary anonymous function.

Instead you can just pass the callback e.g.

.then(callback)

the data, albeit not named specifically ā€˜values’ will be passed to that callback when it is executed.

similarly if you wanted to log that data, instead of

then(values => { console.log(values); }

you could do this instead

.then(console.log)
1 Like

Thank you. I made that update :slight_smile: . Do you see anything else which could benefit from a rewrite to something simpler?

Just to get started

I have gone back to our helper methods.

In your code you import nano

Likewise you could import helper functions in a similar way

// helpers
// could be import { fetchUrl, getJson } from './fetchHelpers.js'
const fetchUrl = (url) => fetch(url)

const getJson = (response) => (
  (Array.isArray(response))
    ? Promise.all(response.map(getJson))
    : response.json()
)

// likewise import { getFirstSplit, getLastSplit, splitOnce } from './stringHelpers.js'

const getFirstSplit = (term, strg) => strg.split(term)[0]
const getLastSplit = (term, strg) => strg.split(term).pop()
const splitOnce = (term, strg) => strg.substring(strg.indexOf(term) + 1)

Using these helpers makes your code a bit more readable down the line

A look at

combineURLData(filteredURLs, options) {
    var finalURLs = [];
    var urlData = Object.entries(filteredURLs);
    urlData.forEach(function(data, index) {
      for(let i=0;i<urlData.length;i++) {
        if(data[0].toLowerCase() === options[i].split("&")[0]) {
          finalURLs.push(data[1]+"&"+options[i].substring(options[i].indexOf("&")+1));
        }
      };
    });
    return finalURLs;
  }

The above is a bit confusing. We are using forEach to loop through urlData, then inside of that we loop through it again. One loop too many?

Instead of this pattern

var result = []

arr.forEach(value => { result.push(value * 2) })

I tend to prefer array’s map, filter or reduce. It saves having the outside factor that we mutate, in other words its self-contained.

In your case there is a conditional as to whether we push a new value or not, so reduce seem the right candidate. It’s worth looking at reduce!

So a first refactor

  combineURLData (filteredURLs, options) {
    const urlData = Object.entries(filteredURLs)

    // Note: we need better names than dataFirst, dataSecond. What are they?
    // They need readable labels.
    return urlData.reduce(
      (finalUrls /* <-- accumulator */, [dataFirst, dataSecond], i) => {
          // options[i] is used a few times, so let's give it an identifier
          const currentOption = options[i]

          // using a helper function instead of options[i].split("&")[0]
          if (dataFirst.toLowerCase() === getFirstSplit('&', currentOption)) {

          // refactoring currentOption.substring(currentOption.indexOf('&') + 1)
          // using a function I have called splitOnce instead (still your code:))
          // all of this can be concatenated using a template string
          // `${javascript here}&${more javascript here}`
          finalUrls.push(`${dataSecond}'&'${splitOnce('&', currentOption)}`)
        }
        return finalUrls
      },
      [] // <-- this is finalUrls initial value. we will push values into this.
    )
  }

combineURLData without comment noise

  combineURLData (filteredURLs, options) {
    const urlData = Object.entries(filteredURLs)

    return urlData.reduce(
      (finalUrls, [dataFirst, dataSecond], i) => {
        const currentOption = options[i]

        if (dataFirst.toLowerCase() === getFirstSplit('&', currentOption)) {
          finalUrls.push(`${dataSecond}'&'${splitOnce('&', currentOption)}`)
        }
        return finalUrls
      },
      []
    )
  }

I have to get this out of the way. I am not saying my refactoring is the way to go. As with many of us, still on the learning path. Hopefully there is something to be taken from it though:)

To be continued…