Forum Discussion
Formula challenge: re-shape and cleanup
I've done some more testing with MAKEARRAY. I've been feeding it large ranges trying to get it to quit but no luck.
Granted, there's no re-shaping being done but Excel is able to calculate this:
=MAKEARRAY(1000000,26,LAMBDA(r,c,INDEX(A1:Z1000000,r,c)))
I'm interested in what part of mtarler 's formula is causing Excel to refuse to calculate. This is presuming it's mostly the presence of MAKEARRAY. Going off the calculation times posted by PeterBartholomew1 in the other discussion, it has to be MAKEARRAY.
Firstly, my congratulations on a very efficient and compact solution.
On the MAKEARRAY discussion, I do not think the function is itself a major problem, provided each calculation is a scalar evaluation. Where I think I hit problems is where the each calculation is actually a full array calc but uses INDEX to restrict the return to a single value. Once each value comes at the price of a full array calc, the time builds as O(n²).
My solution was both more complicated and more expensive than yours (x3) but it did allow me to explore some techniques that I thought might be usable for other problems.
= LET( Fnλ, HSTACK(FirstWordλ,FinalWordλ,Identityλ,Identityλ), baseTbl, WRAPROWS(TOCOL(alumni,1,1),3), extendedTbl, CHOOSECOLS(baseTbl,{1,1,2,3}), u, SEQUENCE(ROWS(extendedTbl),,1,0), Fnsλ, CHOOSEROWS(Fnλ,u), processedTbl, MAP(Fnsλ,extendedTbl,Applyλ), NoBellλ(processedTbl) )The key part of my experiment was to build an array of Lambda functions of the same dimension as the table to which it was to be applied. Then the functions could be applied one by one using MAP. My take is that the approach involves some pretty heavy computational steps but it remains O(n), so should not lock up as the problem size increases.
Thanks for sharing, Peter. I like your approach. It's very clean and easy to follow. MAP has tremendous potential when used with multiple arrays. Later when I'm at my other computer I may extended the rows down to 1 million and have at it with the timer. CHOOSECOLS/CHOOSEROWS/TOCOL/HSTACK appear to be very efficient calculation-wise in handling large arrays.
When I designed this exercise, I was curious how others would go about removing the 'bellman' records. From the other discussion, I recalled SergeiBaklan mention your use of FILTER possibly slowing down your calculation so I went in determined not to use FILTER. Ultimately, I used it thinking it would run smoothly if stored inside the NOBELL LAMBDA and called from my main formula. FILTER seems to be best approach to exclude records at the moment.- Speaking of how people would do the removal of 'bellman' records I noticed in a couple of solution I thought it could be more efficient if that step was moved up earlier. If I read the code right, it looked like it could be done before the splitting of the first and last name in some cases which means that name splitting step would act on a slightly smaller set.
