(Technically, my brain is thinking in terms of PHP, but this theoretically could be any language, so move if appropriate.)

What is a good algorithmic, functional way to navigate the exhaustive permutations of sets, when storing them all in memory simultaneously would be prohibitive?

Here’s my scenario, for the sake of a concrete example.

I have 2 independent sets of 33 arrays. I will pick 5 from the first set, and 5 from the second set, exhaustively.

Each of the arrays is a 4-item array containing effectively a bitmask - so [0,0,0,0] up through [1,1,1,1]. The arrays are known ahead of time, but are not exhaustive of the bitmask - uniqueness is not guaranteed.

An evaluation of the selection of 10 items (2 sets of 5) is done as follows:

For each of the sets chosen, the first item is weighted at 5* , the second at 4*, the third at 3*, the fourth at 2*, and the final at face value.

The selections are then summed by column, so that the result is to turn a 10x4 into a 1x4 result, with the values being in the range 0…30. The index of the highest value in the result is the evaluation of the permutation.

At the end, the idea is to determine the balance of the sets; or in other terms, given a random set of selections, what is the probability that column 0, column 1, column 2, or column 3 is the result?

“But it’s only 33 items.”

Yes, but the permutations of 33 choose 5 is 28 million; and that number squared becomes 811 trillion.