Waiting for a loop to finish using Promises

svoltmer1 · February 15, 2022, 4:58pm

I am using the fetch API to make a RESTfull request to an API endpoint that returns a response with data that I need to iterate over and update properties on the response, after converting to a JSON object. I need the function that updates the properties to complete before passing the new object to a “POST” request to create a new asset on the server based on the new object. Currently, with the code below, I am getting a stack overflow error with the function “getUpdatedBody”, I am sure it has to do with the nature of using Promises with a recursive function. Could someone look this over and let me know if something is incorrect with my syntax or if there is a better way to accomplish this? Thanks!

fetch("http://<my uri>/api/v1/read/page/<site name>/_prototypes/events/event?<credentials>"
  )
  .then(r => r.json())
  .then(data => {
    if(data.success) {
        const currentBody = data.asset;
    
    const getUpdatedBody = async () => {
        for(const prop in currentBody) {
            return new Promise( async (resolve, reject) => {
                if (typeof currentBody[prop] === 'object' && prop != 'structuredDataNodes') {
                    await getUpdatedBody(currentBody[prop]);
                } else 
                    if(prop['body-text']) {
                        prop['body-text'] = 'Body text from API'; 
                } else
                    if(prop['start']) {
                        prop['start'] = '1478059200000';
                } else
                    if(prop === 'name') {
                        prop.name = 'New event from API';
                } else
                    if(prop === 'parentFolderPath') {
                        prop.parentFolderPath = 'news/events/2020/10'
                }
            }); 
        }
    }
        getUpdatedBody().then(newBody => {
            fetch("http:<my uri>/api/v1/create?<credentials>", {
            method: 'POST',
            body: JSON.stringify(newBody)
          })
            .then(r => r.json())
            .then(data2 => {
              if (data2.success)
                console.log('Success');
              else
                console.log('Error occurred when issuing an edit: ' + data.message);
            });
        });
    } else {
      console.log('Error occurred when issuing a read: ' + data.message);
    }
  });

veocode · February 16, 2022, 3:02pm

You have a return inside for-loop, which means that only the first iteration of the loop will be executed and then you immediately leave the function

Try to swap these lines

        for(const prop in currentBody) {
            return new Promise( async (resolve, reject) => {

rpg_digital · February 17, 2022, 1:17am

First off I confess somewhat out of my depth here, especially with recursion and promises, but I have had a go.

Recursion and promise tests

Started with some simpler tasks.

const countdown = (n) => {
    return (n !== 1)
      ? new Promise(
           (resolve, reject) => setTimeout(
             () => resolve(countdown(n-1)), 2000
           )
        )
      : n
}
 
const sum = ([x, y, ...rest]) => {
    return (y !== undefined)
      ? new Promise(
          (resolve, reject) => setTimeout(
            () => resolve(sum([x + y, ...rest])), 1000
          )           
        )
      : x
}

I could be well off here but does this condition equate to dealing with a promise, data to come?

typeof currentBody[prop] === 'object' && prop != 'structuredDataNodes'

walkobj with callback function

I have kind of built the following on that basis, again with a timeOut for simulation. walkObj expects a source object, and a callback function to make any edits.

const walkObj = async(source, fn, parentKey) => {

  const clone = source.constructor()
  
  for (const key of Object.keys(source)) {
    // your condition here possibly?
    if (typeof source[key] === 'object') {
      
      clone[key] = await new Promise(
          (resolve, reject) => setTimeout(
              () => resolve(walkObj(source[key], fn, key)), 2000
          )           
      ) 
    }
    // if the callback doesn't edit the property e.g. returns undefined 
    // the existing property is assigned as is
    else clone[key] = fn(source, key, parentKey) ?? source[key]
  }
  
  return clone
}

Sample object and callback

const person = {
    id: 1,

    personalInfo: {
        name: ['Fred', 'Flinstone'],
        address: {
            road: '222 Rocky Way',
            city: 'Bedrock',
            zip: '70777'
        }
    }
}

// callback for walkobj
const personEdits = (source, key, parentKey) => {
  
  if (parentKey === 'name') {
      return source[key].toUpperCase()
  }
  
  if (key === 'zip') {
      return source[key].replaceAll(/\d/g, 'X')
  }
}

Test

walkObj(person, personEdits)
  .then(console.log) // edited person
  .then(() => console.log(person)) // original person

Output

// edited object
{
  id: 1,
  personalInfo: {
    name: [
      'FRED',
      'FLINSTONE'
    ],
    address: {
      road: '222 Rocky Way',
      city: 'Bedrock',
      zip: 'XXXXX'
    }
  }
}

// source object
{
  id: 1,
  personalInfo: {
    name: [
      'Fred',
      'Flinstone'
    ],
    address: {
      road: '222 Rocky Way',
      city: 'Bedrock',
      zip: '70777'
    }
  }
}

It has been very much a learning exercise, and I’m sure it’s possibly flawed, but maybe there is something to get out of it.

codepen here

windbeneathmywings · February 17, 2022, 2:01am

I would collect all promises in an array and execute Promise.all.

Same concept as fork join in rxjs.

rpg_digital · February 17, 2022, 12:30pm

I did have three things that came to mind whilst working on that code.

Would Promise.all be fitting here
I wonder if RxJS would provide a more elegant solution.
I wonder if @windbeneathmywings has something to say on the matter

When I have time, I will look at how that maybe implemented with a nested structure of promises. In the meantime, any clues?

With regards RxJS, struggling at the moment to find decent tutorials/guides. Most are either 4-5 years old, or if current just reference guides rather than in practice.

svoltmer1 · February 17, 2022, 2:50pm

Thanks for all the time you’ve spent on this! I’ll go back to my original code and see if I can refactor it using some of your ideas. To answer your question, typeof currentBody[prop] === ‘object’ && prop != ‘structuredDataNodes’, deals with the recursion into the JSON object, which has some nested properties that contain arrays of objects. So, to keep calling the “getUpdatedBody” function until all the properties have been iterated over, even those nested within objects.

rpg_digital · February 17, 2022, 5:49pm

So this is just a JSON object, which contains arrays and objects. That’s it?

I don’t see the need on that basis for promises at all. The walkObj function could be simplified to

const walkObj = (source, fn, parentKey) => {
    for (const [key, prop] of Object.entries(source)) {
        source[key] = (typeof prop === 'object')
          ? walkObj(prop, fn, key)
          : fn(key, prop, parentKey) ?? prop
    }
    
    return source
}

Given you are dealing with the resolved object from Response.json(), if you wanted to work with a clone of that object — normally good practice — you could use JSON.stringify and JSON.parse to do that.

const newBody = walkObj(
  JSON.parse(JSON.stringify(currentBody)), // clone currentBody first
  bodyEdits // callback to edit
)

You only seem to be replacing properties based on keys, so I was thinking about a better way of doing that rather than if this else that etc.

You could group those replacements into an object e.g.

const bodyReplacements = {
  'body-text': 'Body text from API',
  'start': '1478059200000',
  'name': 'New event from API',
  'parentFolderPath': 'news/events/2020/10'
}

Your replacement callback function could then just access those props based on a given key

// will return the bodyReplacements property if that key matches
// otherwise undefined
const bodyEdits = (key, prop, parentKey) => {
  // exclude 'structuredDataNodes' here rather than in walkObj
  return (parentKey !== 'structuredDataNodes') 
    ? bodyReplacements[key]
    : undefined
}

My tests with the following data

const body = {
    id: 1,
    structuredDataNodes: { name: 'don\'t touch' },
    nestedData: {
        name: 'xxx',
        someData: {
            start: 'xxx',
            nestedArray: [{parentFolderPath: 'xxx'}]
        }
    }
}

Outputted

// body
{
  id: 1,
  structuredDataNodes: {
    'name': 'don\'t touch'
  },
  nestedData: {
    name: 'xxx',
    someData: {
      start: 'xxx',
      nestedArray: [{parentFolderPath: 'xxx'}]
    }
  }
}

// newBody
{
  id: 1,
  structuredDataNodes: {
    name: 'don\'t touch'
  },
  nestedData: {
    name: 'New event from API',
    someData: {
      start: '1478059200000',
      nestedArray: [{parentFolderPath: 'news/events/2020/10'}]
    }
  }
}

codepen here (Note: you will need to open console)

windbeneathmywings · February 17, 2022, 6:07pm

As you already know I’m an advocate for RxSs. That being said I did not learn by reading a book. I learned by using Angular to build real world applications. Using the Angular RxJs docs, RxJS Website, and whatever info I could find searching the internet for problems encountered. I still follow that same process to overcome issues that I face when using RxJS. That being said the learning curve is an acceptable trade-off for elegant, efficient reactive code avoiding the deep dark depths of callback hell. Some of youtube videos on RxJS are ok as well but I tend to avoid youtube for learning technology unless it is from the source itself like AWS or a conference.

The operator decision tree can be very helpful when starting out but less useful for more complex cases.

https://rxjs.dev/operator-decision-tree

I would highly recommend making RxJS part of any JavaScript / Typescript project.

https://www.npmjs.com/package/rxjs

Reactive programming is also a skill that will set those who are willing to embrace and learn apart from the trash heap of JavaScript developers.

Not to mention many of concepts can be used in other technologies like Apache Kafka an open source streaming platform for building event driven applications.

https://kafka.apache.org/

svoltmer1 · February 18, 2022, 9:54pm

Thanks! I’ve been busy with other projects this week, but I’ll look at this first thing Monday.

rpg_digital · February 19, 2022, 2:15am

Hi @svoltmer1, I know a lot to go through. This has been a bit of a learning exercise for me too.

walkObj utility function

This is an update to the walkobj, an asynchronous version

/**
 * walkObj
 * @param {Object} source - source object to edit
 * @param {Function} callback - callback to make edits
 * @param {Array} excludedKeys - list of excluded keys not to traverse
 * @returns {Object} returns edited clone of source object
 */
const walkObj = function (source, callback, excludedKeys = []) {

    const notExcluded = (key, excluded) => !excluded.some(excludedKey => key === excludedKey)

    const walkObjectEdit = async (source, fn, parentKey) => {

        for (const [key, prop] of Object.entries(source)) {
            source[key] = (typeof prop === 'object' && notExcluded(key, excludedKeys))
                ? await new Promise((resolve, reject) => resolve(walkObjectEdit(prop, fn, key)))
                : fn(key, prop, parentKey) ?? prop
        }
        return source
    }
    // pass in a clone of the source object
    return walkObjectEdit(JSON.parse(JSON.stringify(source)), callback)
}

You can see it takes a source object, a callback to make edits and an array holding a list of properties you don’t want to traverse and edit. So in your case invoked like this.

return walkObj(data.assets, bodyEdits, ['structuredDataNodes'])

fetch, parse and post

Based on your code, this is how I imagine the code would be formatted — obviously I can’t test this.

fetch('http://<my uri>/api/v1/read/page/<site name>/_prototypes/events/event?<credentials>')
    .then(response => response.json())
    .then(function (data) {

        // replaced with methods, which should give a bit more flexibility
        const bodyReplacements = {
            'body-text': (text) => 'Body text from API',
            'start': (start) => '1478059200000',
            'name': (name) => 'New event from API',
            'parentFolderPath': (path) => 'news/events/2020/10'
        }

        // will only call the method ?.(prop) if the key exists or return undefined
        const bodyEdits = (key, prop, parentKey) => bodyReplacements[key]?.(prop)

        return walkObj(data.assets, bodyEdits, ['structuredDataNodes'])
    })
    .then(newBody => {
        return fetch('http:<my uri>/api/v1/create?<credentials>', {
            method: 'POST',
            body: JSON.stringify(newBody)
        })
    })
    .then(response => response.json(response))
    .then(data => {
        console.log(
            (data.success)
                ? 'Success'
                : `Error occurred when issuing an edit: ${data.message}`
        )
    })
    .catch(console.log)

Simple test using jsonplaceholder

This is a test I did with jsonPlaceholder and it appears to function correctly.

fetch('https://jsonplaceholder.typicode.com/users')
    .then(response => response.json())
    .then(function (users) {

        const userReplacements = {
            name: (name) => name.toUpperCase(),
            zipcode: (zip) => zip.replaceAll(/\d/g, 'X'),
            phone: (number) => 'Call Me',
            lat: (latitude) => latitude.replaceAll(/\d/g, 'X')
        }

        const userEdits = (key, prop, parentKey) => userReplacements[key]?.(prop)

        return walkObj(users, userEdits, ['company', 'geo'])
    })
    .then(newUsers => console.log(JSON.stringify(newUsers, null, 2)))
    .catch(console.log)

codepen here

Will be interested to see how you get on

dennisjn · February 19, 2022, 2:03pm

I am wondering why you are using async without any await.
I wonder if the second async should actually be the await.
I’m fairly certain that using async without the await achieves nothing; but I could be wrong.

rpg_digital · February 19, 2022, 6:27pm

Its there.

? await new Promise((resolve, reject) => resolve(walkObjectEdit(prop, fn, key)))

svoltmer1 · February 22, 2022, 7:03pm

Okay, almost thought I had this using some of your code, but apparently the response.data takes on a different way of labeling properties on one of the nested arrays. I’ve uploaded the data.json file so you can see what I am talking about. There is a property “structuredData.structuredDataNodes” which is an array of objects with a property named “identifier”, this property will need to be checked for the values named “body-text”, “start”, “end” and then if found update the corresponding “text” property of that object with the supplied value which I would like to supply in the “propertyEdits” object like your “bodyReplacments” object. I not certian how to accomplish this though.

rpg_digital · February 22, 2022, 7:36pm

So inside structuredDataNodes, are you replacing ‘body-text’, ‘start’ with the same corresponding values e.g. ‘Body text from API’, ‘1478059200000’

If so in theory you could just make the following change to the code. Note you would need to add an ‘end’ method as well this time.

const bodyReplacements = {
    'body-text': () => 'Body text from API',
    'start': () => '1478059200000',
    // add end in here as well
    'end': () => 'replacement value here'
    'name': () => 'New event from API',
    'parentFolderPath': () => 'news/events/2020/10'
}

// can stay the same
const bodyEdits = (key, prop, parentKey) => bodyReplacements[key]?.(prop)

// remove the third argument excluding structuredDataNodes e.g. ,['structuredDataNodes']
return walkObj(data.assets, bodyEdits)

If the edits are different for structuredDataNodes, then we can deal with that. Note bodyEdits passes in (key, prop, parentkey), so we can check to see if the parent is ‘structuredDataNodes’ and serve something else up.

I am sorry, I have to pop out for 30 minutes. Will be back

rpg_digital · February 22, 2022, 10:36pm

Here is a version with a bit more fine-grained control.

The bodyReplacements object now has a parent object which you can use to single out selected parent properties like ‘structuredDataNodes’ and replace their child properties separately.

Otherwise if the current parent doesn’t match one of the selected parent keys then we go to the child object instead for some default property edits.

If at a later date you want to make unique edits to say ‘metaData’, you can just add those to the bodyReplacements’ parent object.

.then(function (data) {

    const bodyReplacements = {
        // edits with a matching parent key
        parent: {
            // in this case 'structuredDataNodes'
            structuredDataNodes: {
                'body-text': () => 'new body text value',
                'start': () => 'new start value',
                'end': () => 'new end value'
            }
        },
        // default child replacements
        child: {
            'body-text': () => 'Body text from API',
            'start': () => '1478059200000',
            'name': () => 'New event from API',
            'parentFolderPath': () => 'news/events/2020/10'
        }
    }

    // walkObj callback
    const bodyEdits = (key, prop, parentKey) => {

        const replaceFrom = (parentKey)
            ? userReplacements.parent[parentKey] || userReplacements.child
            : undefined

        return replaceFrom && replaceFrom[key]?.(prop)
    }

    return walkObj(data.assets, bodyEdits)
})

Body edits

A breakdown of the new bodyEdits callback function

const replaceFrom = (parentKey)

Do we even have a parentKey?

? userReplacements.parent[parentKey] || userReplacements.child

If so replaceFrom will be assigned either a matching parent property e.g. parent.structuredDataNodes or the default child object

: undefined

If there is no parent key it will be assigned undefined

return replaceFrom && replaceFrom[key]?.(prop)

We then return either undefined or try to match the current key, say for instance ‘body-text’ and if it has that property call the function passing in it’s current value.

If we don’t find the current key in our replacements, say ‘createdBy’ we just return undefined

Note I am making use of the logical And operator here (&&) which if replaceFrom is undefined will return undefined otherwise it will return the right hand side. It might be better to be a little less cryptic, but will leave it for now.

const x = undefined
const y = 'A string'
const z = {} // empty object

x && 'this value' // undefined
y && 'this value' // 'this value'
z && 'this value' // 'this value'

Oh and a codepen demonstrating it in action. In this example I wanted the ‘name’ property inside ‘company’ to receive a unique value to the other ‘name’ properties.
codepen here

svoltmer1 · February 23, 2022, 2:59pm

Here is the issue. Once I iterate to the structuredDataNodes property, the value is an array of objects, each object has a parameter named “identifier” with a value of either body-text, start, end, etc…, but I need to change the value of the corresponding “text” property for those objects. Let me know if you can see what I am referring to from my previous post’s screen capture. Thanks!

rpg_digital · February 23, 2022, 3:09pm

Ah ok, I missed that, so the parentKey is going to be an index of the array. hmmm. Will have to come back to you.

rpg_digital · February 23, 2022, 6:36pm

Unfortunately @svoltmer1 I don’t have access to the complete data and all the requirements, which is making this a bit difficult with the brief being somewhat drip-fed. If I had the complete picture I could maybe think of a better strategy.

Going on what you have given:

I have made some alterations to walkObj:

I am no longer passing around the parentKey, but instead passing in the source object or you could say parent to the callback instead. This will give us access to sibling properties.
I have stripped out the exclude functionality as we don’t appear to be needing that now.

/**
 * walkObj
 * @param {Object} source - source object to edit
 * @param {Function} callback - callback to make edits
 * @returns {Object} returns edited clone of source object
 */
const walkObj = function (source, callback) {

    const walkObjectEdit = async (source, fn) => {

        for (const [key, prop] of Object.entries(source)) {
            source[key] = (typeof prop === 'object')
                ? await new Promise((resolve, reject) => resolve(walkObjectEdit(prop, fn)))
                : fn(key, prop, source) ?? prop // source/parent passed in instead
        }
        return source
    }
    // pass in a clone of the source object
    return walkObjectEdit(JSON.parse(JSON.stringify(source)), callback)
}

On to the edits:
Inside of assetEdits in the code below I am checking to see if the key equals ‘text’ and whether it has a sibling of ‘identifier’. I do this using ‘parent.identifier’

If it does I look up the keys in assetReplacements and see if one matches the identifier value e.g. ‘start’. If a match is found the method is invoked and a value returned.

Otherwise we do as we have been doing and just try and find a method inside assetReplacements matching the given key and invoke it.

Here is the code.

/**
 * walkObj
 * @param {Object} source - source object to edit
 * @param {Function} callback - callback to make edits
 * @returns {Object} returns edited clone of source object
 */
const walkObj = function (source, callback) {

    const walkObjectEdit = async (source, fn) => {

        for (const [key, prop] of Object.entries(source)) {
            source[key] = (typeof prop === 'object')
                ? await new Promise((resolve, reject) => resolve(walkObjectEdit(prop, fn)))
                : fn(key, prop, source) ?? prop
        }
        return source
    }
    // pass in a clone of the source object
    return walkObjectEdit(JSON.parse(JSON.stringify(source)), callback)
}

fetch('http://<my uri>/api/v1/read/page/<site name>/_prototypes/events/event?<credentials>')
    .then(response => response.json())
    .then(function (data) {

        const assetReplacements = {
			'body-text': () => 'Body text from API',
			start: () => '1478059200000',
			end: () => '3762059200000',
			name: () => 'New event from API',
			parentFolderPath: () => 'news/events/2020/10'
		}

		const assetEdits = (key, prop, parent) => {
			return (key === 'text' && parent.identifier)
				? assetReplacements[parent.identifier]?.(prop)
				: assetReplacements[key]?.(prop)
		}

        return walkObj(data.assets, assetEdits)
    })
    .then(newBody => {
        return fetch('http:<my uri>/api/v1/create?<credentials>', {
            method: 'POST',
            body: JSON.stringify(newBody)
        })
    })
    .then(response => response.json(response))
    .then(data => {
        console.log(
            (data.success)
                ? 'Success'
                : `Error occurred when issuing an edit: ${data.message}`
        )
    })
    .catch(console.log)

A codepen test using similar data.

svoltmer1 · February 23, 2022, 7:14pm

Thanks for working with me on this! I can give you more information on the scope of this project. Simply put I use a content management system that has a REST API endpoint for various actions within the CMS. We use SharePoint internally and when events are created there, I need to send the data from the SharePoint event as a GET request to the CMS API to create a new event within the CMS based on another event asset within the CMS, of which I am reading in first, then altering the needed properties from the SharePoint event and then issuing a POST to the CMS API to create the new event asset from the base event asset with the modified properties from the SharePoint event. I know that’s a lot, but you asked for it. LOL!

In your code, you have assetReplacements[parent.identifier]?.(prop), but that will be assetReplacements[‘body-copy’]?.(prop) and I need to modify the “text” sibling property value. Am I incorrect?

rpg_digital · February 23, 2022, 7:43pm

No from what I can tell you should be able to leave the code as it is . Have a look at the codepen and check out the console results on the test I did. The text property is being modified based on the sibling ‘identifier’.