Question on presenting stat weights.

[Caltiom]

(03-10-2012 10:46 AM)Kor Wrote:  I've actually been considering that very question closely lately. As Laguna has pointed out to me at least once, despite the fact that goal points are objectively better metrics, they are much more difficult to compute, thus making them somewhat less practical. That said, stat weights must be calculated for each individual character and gear setup. Goal points (or better yet, an interpolated function gained from a series of goal points at varying itemization amounts) are universal based solely on class and spec. I've been contemplating hijacking Simcraft (or making a module for it) to calculate goalpoints over a range of itemization values by iteratively adjusting the stats until the difference in values between the specified stats are below a certain margin. It would be very computationally intensive, but then again, extreme accuracy isn't really needed (getting stats within ±10 of the actual goal point should be sufficient), and the calculations would only need to be performed once per spec per patch. This would also segue rather nicely with my current Numerical Computation class (may use it as the class project, come to think).

It would also make finding BIS gearsets much easier. Basically, iterate the modules until the averaged delta of all the stats off of the goalpoint line is minimized, and you should have BIS (and considering there's only 6-8 slots with mods in this, this is pretty easy to do even manually).

Sounds very interesting Kor, that would be a very great feature to have!

One problem in practice will be that it might not be feasible to calculate goalpoints where all stats ( including willpower and power ) are included. Because willpower/power have such inflexible marginal returns, always at a quite high level, I think you will get goalpoints which will not match any given gear permutation.

Your algorithm looks great Kor. I'd say I can give you a scale factor with 5% accuracy, with centered scale delta ( which means more or less nearly zero inherent error to a true derivative. Thanks again LagunaD for that ) from -15 to +15 with around 15seconds cpu time on a Intel i5 with 4 threads. ( Well if I did my calculations correct ).
Casey and myself could help you by revamping our scale factor calculation a bit, making it more like a internal API in SimC. ( It's currently a little bit of a mess, very old code )

But to also contribute something to the original topic:
+1 scale factors are really not that feasible, even if you have a more smooth distribution ( which you don't seem to get with your healing simulation. Great work btw! ).
In practice you'll always switch mods with +30 or more stats, so I don't see any problem in using those higher deltas.

What seems missing here is a statistical error for those scale factors. If you can compute that ( Central Limit Theorem ) you can estimate how much of the variation is due to statistical error ( at least with some confidence ) and wheter the whole problem ( highly osciliating scale factors ) is truely a inherent problem of a healing model, or if most of it is only statistical noise.

[RuQu]

(03-10-2012 05:28 PM)Caltiom Wrote:  But to also contribute something to the original topic:
+1 scale factors are really not that feasible, even if you have a more smooth distribution ( which you don't seem to get with your healing simulation. Great work btw! ).
In practice you'll always switch mods with +30 or more stats, so I don't see any problem in using those higher deltas.

What seems missing here is a statistical error for those scale factors. If you can compute that ( Central Limit Theorem ) you can estimate how much of the variation is due to statistical error ( at least with some confidence ) and wheter the whole problem ( highly osciliating scale factors ) is truely a inherent problem of a healing model, or if most of it is only statistical noise.

Thanks. It's been a fair amount of effort, but a good learning exercise and some people are finding it useful even in its early stages. Always room to improve and learn more though.

I know your work is mostly with the swtor port of SimulationCraft. There is some jargon here I am not familiar with.

By +1 scale factors, do you simply mean increasing a stat by 1?

The word delta, which I assume has its traditional meaning of "delta X" is "change in X," spends a lot of time naked. It seems to be used multiple times in reference to different variables without ever specifying the variable that is changing. This might make great short-hand between collaborators, but it makes it hard for an outsider to follow.

For the Combat Medic and Bodyguard healing, there is no RNG when using the Expected Heal value to average out the crits and the min-max range. The rotation for a given set of stats and a given boss DPS and swing_timer will always be the same. However, there is variability between the results for similar stats. Any advice on how to put error bars on that? This troubles me a fair amount.

For Scoundrel healing and Commando Gunnery DPS, there is RNG involved in assorted procs. I don't tend to do much with calculating statistics manually.

It looks like I would need to store all of the HPCT or DPS values, instead of keeping a running sum, and then find the mean (which I currently do), and also the difference from the mean for each, and square that difference to get the variance. The CLT then says the variance of the distribution would be the average of those variances?

In practical terms, what does this gain me? Does it simply allow me to define a loop that computes the running variance on every pass and continue to iterate until the variance was below a threshold or (to prevent infinite loops) the number of iterations hit some limit? This could save time if it converges quickly, and if it doesn't I'm exactly where I am now with a fixed number of rotations. It also allows me to put a number on the uncertainty, but most users won't care what that value is. Please don't take this the wrong way...I'm just honestly curious what I gain in exchange for the increased overhead, and also how to best go away implementing such a change.

[Caltiom]

Sorry for the SimC jargon.

By scale factor I mean either a partial derivate of ∂( dps ( stat_n ) / ∂( stat_n ) or the differential quotient dps ( stat_n + delta ) - dps ( stat_n ) / delta ( which approaches the partial derivate when delta -> 0 ).

The delta usually references to this delta used for calculating the scale factor, gear weight or whatever you want to call it.

If you don't have any rng in your simulation, and the dps of each iteration is equal to that of any other, then there is no gain in calculating statistical error, as it will be 0.

So for those modules where you don't have statistical variation, and when going from 0 to +1 of a certain stat you even get a loss of hps, then that's just how it is ( within your model, at least ). You have to live with it, and if your model really is a correct one of the actual gameplay of a healer, than thats just it and a +1 stat increase means a hps loss.
This then goes back to the question for what you want your scale factors, what is the purpose of them? If you want to answer the exact question of how much gain +1 of a stat for exactly this gear set is, then everything is fine.
If you want a more pracital relevance try using a bigger delta, smooth out the curve ( I'm sure Kor can help you out here, maybe http://en.wikipedia.org/wiki/Polynomial_regression ? I haven't paid much attention to numerical mathematics )

If you do have variation, then yes you need to collect the dps of each iteration, calculate the mean. then you get variation = 1 / ( iterations - 1 ) * sum ( ( dps( iteration_i ) - dps_mean ) ) ^2 ). The variation of the mean then is variation_mean = variation / #iterations. And then stddev_mean = sqrt( variation_mean ), and with a 95% confidence the true mean will be within [ mean - stddev_mean * 1.96, mean + stddev_mean * 1.96 ].

Code:

# Some C++ example code here:
double[#iterations] dps_results;

calculate_dps( dps_results ); // do your simulation/calculation, and save each dps result into the dps_results array

double dps_mean = 0;
for ( int i = 0; i < #iterations; i++ )
{
dps_mean += dps_results[ i ];
}
dps_mean /= #iterations;

double dps_variation = 0;
for( int i = 0; i < #iterations; i++ )
{
dps_variation += ( dps_results[ i ] - dps_mean ) ^ 2;
}
dps_variation /= ( #iterations - 1 );

double dps_mean_variation = dps_variation / #iterations;
double dps_mean_stddev = sqrt( dps_mean_variation );

double dps_95%confidence_error = dps_mean-stddev * 1.96;

-> true dps lies between  dps_mean - dps_95%confidence_error and dps_mean + dps_95%confidence_error

As you see it converges by sqrt( #iterations ), or lets say quite slow.
Oh and we at SimC just let the user specify a number of iterations so he can choose what he wants ( it's his cpu time after all ^^ ).


What you gain from it? Well you can quantify the statistical error to see how much of the variation arises because of it. I can't say if you'd gain a lot of it. If you only rng some procs and not much else, your statistical variation will probably extremly small and you can skip the whole thing.
In SimulationCraft we use rng for nearly everything ( except for base-damage variation, at least with default settings ), and thus analyzing the statistical error is essential to the whole concept of the simulator.

[Kaedis]

Quote:Let me make sure I am reading this (completely fictional) graph correctly. It plots the points where EP_Power ~= EP_Crit. So at 200 Crit and ~55 they are approx. the same, and the optimal point is ~ (200,300).

The curve describes the best combination of Power and Crit given verious levels of P+C, yes. So at 200 stat budget, for example, P = 32.65, while C = 167.34. The curve is probably best thought of as a parametric curve, with t = P + C.

Quote:Now, when you say "increase your itemization total by a step amount," you mean to increase the stat budget. So find n such that EP_Crit ~= EP_Power for Power+Crit = 200, then find n again for P+C = 225.

The set amount on budget isn't where it gets close. The error value on that refers to getting the calculated balance point (P,C) within a radius of 2 stat points of it's actual value.

Quote:6 simulations per point. Do you mean instead of doing 600/5 = 120 simulations you do 600/100 = 6 simulations with Power = BasePower + 100*n, Crit = BaseCrit + 100*(6-n)? How does this get so close as + or - 2? Or am I misinterpreting?

Sorry, I was actually off slightly on this. It would take ~60 to calculate it linearly, and 9 to calculate via a binary search.

The reason it only takes 6 is because using a binary sort algorithm reduces runtime to Log(N) rather than N. Example, using a linear search, we need to perform a a search for the breakpoint somewhere over a range of 600. This is a single step on the budget curve calculation, representing the point where Budget = 600, but we need to calculate the various steps at that budget point as we shift that budget from being entirely in crit to being entirely in Power. Normally, this would take an average of about N/2 simulations, or somewhere around 60. However, with a binary algorith, it would only take 6 to get the error down under 2.5 (which is the error bound you would have assuming you calculated linearly with step size 5).

On the first calculation, you're checking the both end-points (2 calculations). Now, you confirm that the end points have a cross somewhere between them (ie. the EP of crit > power at one side, opposite at the other), and if now, pick the end with the smaller difference between EPs. If there is a cross between them, calculate the midpoint and check which half of the range the cross occurs in. Continue splitting the data until you've dropped under the specified error margin. This is where the Log(n) comes in. See, after the end-point calculation, over this range, your maximum error is 300. After the second, it's 150, and the third, 75, the fourth 37.5, then 18.75, 9.375, 4.6875, and finally 2.34375 on the 9th calculation. Regardless of where the crossover is in that range, it will take exactly 9 calculations to find it to within error < 2.5, whereas the linear search takes between 2 and 121 calculations, averaging to about 60.

Quote:Let me try and put this in code, as that is generally easier to follow than words...

Could we not simplify this further, and have the user provide the amount of moddable Power/Crit/Surge/Alacrity on their gear? We would then only need to do one budget loop, instead of form a curve.

The point of the algorithm is that the user doesn't need to do the calculations, using an application or otherwise. This optimization line is dependent only on spec and class, not on gear. In fact, the entire point of the algorithm is to find the optimal stat points along all possible budgets of gear. I'll try to throw together some pseudocode of what I'm thinking sometime today.

[RuQu]

(03-10-2012 06:38 PM)Caltiom Wrote:  Sorry for the SimC jargon.

No worries. Such is the nature of jargon. It makes communication easier...right up until a new guy walks in the room.

Quote:By scale factor I mean either a partial derivate of ∂( dps ( stat_n ) / ∂( stat_n ) or the differential quotient dps ( stat_n + delta ) - dps ( stat_n ) / delta ( which approaches the partial derivate when delta -> 0 ).

Is that second one supposed to be: [dps ( stat_n + delta ) - dps ( stat_n ) ]/ delta ?

Quote:The delta usually references to this delta used for calculating the scale factor, gear weight or whatever you want to call it.

And in the above, delta is how much you change stat_n by?

Quote:If you don't have any rng in your simulation, and the dps of each iteration is equal to that of any other, then there is no gain in calculating statistical error, as it will be 0.

That's what I thought.

Quote:So for those modules where you don't have statistical variation, and when going from 0 to +1 of a certain stat you even get a loss of hps, then that's just how it is ( within your model, at least ). You have to live with it, and if your model really is a correct one of the actual gameplay of a healer, than thats just it and a +1 stat increase means a hps loss.

I found the source of my "wiggle." As I was testing and experimenting trying to find a better metric, I eventually settled on not counting the free filler ability cast time, but counting the healing it does, since I decided that when the tank is at 98%, that becomes "optional healing" and you wouldn't waste a real heal on it.

I then kept testing...and forgot to turn the HS time back off in the cast_time_counter when I made the plots. So I was plotting true HPCT, not HPRCT. And, the algorithm was (correctly) deciding to use the low HPCT but high HPR (Healing per Resource) fillers to top off the tank. This correct choice of a low HPCT ability lead to a drop in the curve as stats improved, which makes little sense intuitively.

Compare:
HPCT Chart in Rakata
HPRCT chart in Rakata.

The proper del_HPRCT vs del_Stat (Stat Weight) chart.

Quote:This then goes back to the question for what you want your scale factors, what is the purpose of them? If you want to answer the exact question of how much gain +1 of a stat for exactly this gear set is, then everything is fine.
If you want a more pracital relevance try using a bigger delta, smooth out the curve ( I'm sure Kor can help you out here, maybe http://en.wikipedia.org/wiki/Polynomial_regression ? I haven't paid much attention to numerical mathematics )

If you do have variation, then yes you need to collect the dps of each iteration, calculate the mean. then you get variation = 1 / ( iterations - 1 ) * sum ( ( dps( iteration_i ) - dps_mean ) ) ^2 ). The variation of the mean then is variation_mean = variation / #iterations. And then stddev_mean = sqrt( variation_mean ), and with a 95% confidence the true mean will be within [ mean - stddev_mean * 1.96, mean + stddev_mean * 1.96 ].

Code:

# Some C++ example code here:
double[#iterations] dps_results;

calculate_dps( dps_results ); // do your simulation/calculation, and save each dps result into the dps_results array

double dps_mean = 0;
for ( int i = 0; i < #iterations; i++ )
{
dps_mean += dps_results[ i ];
}
dps_mean /= #iterations;

double dps_variation = 0;
for( int i = 0; i < #iterations; i++ )
{
dps_variation += ( dps_results[ i ] - dps_mean ) ^ 2;
}
dps_variation /= ( #iterations - 1 );

double dps_mean_variation = dps_variation / #iterations;
double dps_mean_stddev = sqrt( dps_mean_variation );

double dps_95%confidence_error = dps_mean-stddev * 1.96;

-> true dps lies between  dps_mean - dps_95%confidence_error and dps_mean + dps_95%confidence_error

As you see it converges by sqrt( #iterations ), or lets say quite slow.
Oh and we at SimC just let the user specify a number of iterations so he can choose what he wants ( it's his cpu time after all ^^ ).


What you gain from it? Well you can quantify the statistical error to see how much of the variation arises because of it. I can't say if you'd gain a lot of it. If you only rng some procs and not much else, your statistical variation will probably extremly small and you can skip the whole thing.
In SimulationCraft we use rng for nearly everything ( except for base-damage variation, at least with default settings ), and thus analyzing the statistical error is essential to the whole concept of the simulator.

Yeah, I only RNG procs, and nothing else, so I will probably skip adding in the uncertainty for now, but keep it in mind for later. Especially if the variance from procs is as low as you suspect, I think I can probably score some gains by using it as a factor for stopping before the specified number of iterations are reached which should save some time.

As for choice of deltas, the minimum change people can likely make will be on the order of 30. These curves will also change as gearing changes, so there is no compelling reason to show the change over too large of a window. I could cut the time down a fair amount by switching to intervals of 15, and then switching the window to -150 to +150.

While continuing to work on this optimal curve concept, in the interim I can do +1 stat weights by doing +15/15, since the effects should be linear on a local enough scale. That should eliminate some of the possibility of negative values for +1 stats.

[RuQu]

(03-10-2012 11:29 PM)Kor Wrote:  The reason it only takes 6 is because using a binary sort algorithm reduces runtime to Log(N) rather than N. Example, using a linear search, we need to perform a a search for the breakpoint somewhere over a range of 600. This is a single step on the budget curve calculation, representing the point where Budget = 600, but we need to calculate the various steps at that budget point as we shift that budget from being entirely in crit to being entirely in Power. Normally, this would take an average of about N/2 simulations, or somewhere around 60. However, with a binary algorith, it would only take 6 to get the error down under 2.5 (which is the error bound you would have assuming you calculated linearly with step size 5).

On the first calculation, you're checking the both end-points (2 calculations). Now, you confirm that the end points have a cross somewhere between them (ie. the EP of crit > power at one side, opposite at the other), and if now, pick the end with the smaller difference between EPs. If there is a cross between them, calculate the midpoint and check which half of the range the cross occurs in. Continue splitting the data until you've dropped under the specified error margin. This is where the Log(n) comes in. See, after the end-point calculation, over this range, your maximum error is 300. After the second, it's 150, and the third, 75, the fourth 37.5, then 18.75, 9.375, 4.6875, and finally 2.34375 on the 9th calculation. Regardless of where the crossover is in that range, it will take exactly 9 calculations to find it to within error < 2.5, whereas the linear search takes between 2 and 121 calculations, averaging to about 60.

Let's see if I can put this in order.

Code:

//Set a budget. ie P+C= 600;
budget = 600;
errStat = 2.5; //max allowable error in optimal stat
//Calculate 2 points in function DPS(P,C)
P1=0;
C1=budget;
P2=budget;
C2=0;
while(ABS(P1-P2)>errStat){
    DPS_1 = DPS(P1,C1);
    DPS_2 = DPS(P2,C2);
//Identify the midpoint:
    Pm = (P1+P2)/2;
    Cm = (C1+C2/2);
//Calculate midpoint DPS:
    DPS_m = DPS(Pm, Cm);
//Compare:
    if(ABS( DPS_m-DPS_1) < ABS( DPS_m-DPS_2))
    {
        P2 = Pm;
        C2= Cm;
    }else{
        P1=Pm;
        C1=Cm;
    }
}

Quote:The point of the algorithm is that the user doesn't need to do the calculations, using an application or otherwise. This optimization line is dependent only on spec and class, not on gear. In fact, the entire point of the algorithm is to find the optimal stat points along all possible budgets of gear. I'll try to throw together some pseudocode of what I'm thinking sometime today.

Currently I assume the obvious skills that all specs must take for their role, and allow the user to set the other skills as they want, allowing them to get a feel for how much losing a point in one skill, or taking another will change their output.

Since the arbitrary budget optimal curves would have to be pre-determined for every spec, I would either need to decrease the amount of choice in specs, or generate lots of pre-determined curves every patch and hardcode them in so every choice of settings referenced the appropriate one.

Of course, I may not have a choice here, as the time to run the optimal distribution for even one budget on a class with procs (and therefore numerous iterations averaged) might be excessive.

Am I correct to say that the idea would be to generate a curve for a budget of (say) 200-600. Then the user enters their stats, and says "My crit/power budget is 300." At this point the only computation the stat calculator has to do is check a vector for the entry where P+C = 300 and report the values of P and C? No need to run the simulation at all?

One possibility that occurs to me: the user can set their skills as they want, and experiment with the Calculate DPS button. Once they settle on their spec, they have a button which says "Compute and Store Optimization." Pressing that takes a LONG time, but will generate their spec's optimal curve, and store it in a local file. When clicking on "Compute Stat Weights" it would load the curve from this file. Pre-done curves for select specs could be made available for download as they become available, but I would not be responsible for generating huge numbers of curves for every possible option.

[Kaedis]

Quote:Currently I assume the obvious skills that all specs must take for their role, and allow the user to set the other skills as they want, allowing them to get a feel for how much losing a point in one skill, or taking another will change their output.

Since the arbitrary budget optimal curves would have to be pre-determined for every spec, I would either need to decrease the amount of choice in specs, or generate lots of pre-determined curves every patch and hardcode them in so every choice of settings referenced the appropriate one.

Of course, I may not have a choice here, as the time to run the optimal distribution for even one budget on a class with procs (and therefore numerous iterations averaged) might be excessive.

Actually, you probably don't need to recalculate or pre-code the possibilities in. Keep in mind that A) we're only calculating these stats to a precision of about ±2.5, or around ±0.5% - 1.5%, and B), users are unlikely to be able to, on average, get within more than a quarter the size of the step of the mod slot (so, for crit and power, that step is +37 of the stat, so on average a user get get their crit and power to agree with the curve to within ~9).

That's a fairly significant margin of error, objectively, but that's ok, since people aren't looking for the exact stats to have, just the goal points that they can use to balance towards. The idea is that given an arbitrary about of both of the paired stats, the program, knowing the optimization curve for those paired stats, can tell the user where the balance point is and how to get closer to it, if possible. High precision isn't needed, because the system itself limits the precision (and the net benefit of having more precision is almost nil).

Beyond that, the addition or subtraction of "optional" talents is by definition unlikely to have a sizable effect on dps, and even where it does, it's even less likely to have a noticeable effect on the stat balance points (in fact, given our precision limit, I'm pretty certain that statistical noise is several orders of magnitude larger than any effect optional talents could have on the balance points).

Quote:Am I correct to say that the idea would be to generate a curve for a budget of (say) 200-600. Then the user enters their stats, and says "My crit/power budget is 300." At this point the only computation the stat calculator has to do is check a vector for the entry where P+C = 300 and report the values of P and C? No need to run the simulation at all?

Yes, this is the idea. The P+C curve will assuming that the user has an optimized A+S stat balance already (or is balancing that at the same time), and that the ratio of the user's A+S budget to P+C budget is consistent with the ratio found on the specified gear-level. Once again, though, and error from the assumptions is most likely to be well below the level of actual inaccurate results. For example, let's say that the actual balance point for 500 budget is 322.52 Crit and 177.48 Power. Let's say the program has identified this balance point as 325±2.5 Crit and 175±2.5 Power. Now, through assumptions on the curve and such, let's say the program is recommending the user be at 157 Power and 343 Crit (18 off of the calculated balance point), as opposed to the other option of 194 Power and 306 Crit (19 off of the calculated balance point). Objectively, the program has "done it wrong", as the program is recommending gearing that is actually 20.48 off the true balance point, over a gearing that is only 16.52 off. However, note that the net difference is that only 3.96 stats are "misplaced", outside of optimal distribution, and that at this point on the curve (barely 5 times that off the true optimization point), that the value of Crit and Power are still very nearly equal, and thus the net dps loss from that 3.96 incorrect stat placement is very close to zero.

Heck, we could even build in a mini character creater (maybe I can work with AskMrRobot about a plug-in to their character builder, or an import option) that accounts for the actual budget ratios on the gear (for example, what if the person in question isn't using custom a wrist and belt slot? What if they are using the 61 Primary / 11 Power/Crit mods instead of the generally-superior 48 Primary / 37 Power/Crit ones?), and recommend specific module changes based on that character's stats and current module setup.