+- SWTOR Mechanics Forums (http://mmo-mechanics.com/swtor/forums)
+-- Forum: General (/forum-1.html)
+--- Forum: Game Mechanics (/forum-5.html)
+--- Thread: Softcaps discussion (/thread-659.html)
RE: Softcaps discussion - Gorlough - 03-13-2012 05:49 PM
Do get numbers to work with, you basically have to rule out variables.
Theorycrafting does this for you. It breaks down the game mechanics to the most basic formulae to start with.
To get more accurate results, you then go into simulation with those results.
Simulation adds variables again (heat management i.e.) and presents you with more accurate data, altered to the new circumstances.
The first step in simulating is to include the very basic mechanics of your class. This will usually end with a number on DPS, HPS or Mitigation.
The next step in simulating is to add even more variables (a known strawman mob i.e.).
The last step would be to integrate a RNG.
RNG inclusion is key to certain mechanics (i.e. is my tank simulation able to survive hard crits?).
This is then the point to compare to (multiple) parses. If the numbers match, your simulation was accurate and covered almost every aspect of that fight mechanic. If they don't match - your assumptions for the simulation were wrong.
But you have to start at a point. And it's end the theory end of things.
You don't saddle a horse from upside down, you know?
RE: Softcaps discussion - Caltiom - 03-13-2012 05:49 PM
The old discussion about simulators and combat logs
You certainly have a point about our confidence in drawing conclusions from certain assumptions. While Kor really gets tired of mentioning the ability to draw conclusions from deterministic test ( usually through the floating combat text ), this point isn't valid for mechanics which aren't inherently not deterministic ingame as well.
So your point about the accuracy cap is quite valid. And if we didn't have any ingame character sheet showing us information about crit chance, the whole thing would become extremly uncertain. We still can't validate if the actual crit chance is equal, or in any means correlates to the one displayed in the character sheet. We just assume it does, but we can of course trust this assumption much more than for example the 8% accuracy cap.
But besides that accuracy cap, there is nearly no other non-deterministic problem suspect to inherent distrust. There is no reason to believe the base-damage between min and max values is not evenly distributed, etc.
Then about simulators: You seem to mix Rawr into the Simulation family here. While Rawr did have some simulation towards the end, it's basically a formulator with very complex aggregated formulas used for calculating a deterministic dps.
I personally really do believe ( each time I encounter such a discussion, a bit stronger ) in the much bigger advantage of simulations being able the reduce a complex problem to its simple puzzle-stones instead of having to create 1 complex formula for solving the whole problem deterministically. This advantage vastly outsets the disadvantage of not being able to deterministically and cheaply ( in terms of cpu time ) calculate the result.
You mention peer reviewed journals. It would be great if we could engage someone to do that, unfortunately we don't have the money for it
But at least SimulationCraft ( and many others, unfortunately not all ) is completly transparent. You can always check the class module source code ( which can be understood by a 10 year old without much coding experience ) and verify the implementation. We try to add as many comments directly to the code as possible, to make things comprehensible.Unfortunately there is never that much code review, and we invite everyone here to do so to improve the simulation model. While there are a lot of ways to track the output and potential errors/problem of a ability in SimC, it would often have been easier if the code had been reviewed and human errors had been spotted earlier.
Then there is always the discussion of "what metric" and what "situation model" is even important to improve ones character? While the DPS & Standstill model is the classic one, even damage dealers now try to look at a lot of different metrics and try to model movement fights and others. For Healer theorycrafting tools, this discussion is even more important and it is very hard to decide which basket of measurements is important for them.
But I don't want to go into more detail about this discussion here. Just mentioning that at least in SimC there are not many technical boundaries to simulating very complex fights. The enemy is just a player extension and you can model him with many complex mechanics. Healer healing a Tank getting hit on the head by the Enemy is easily possible. There are .simc files available on the wow project trying to emulate certain very specific boss fights, and while there are certainly boundaries for all of this, 0% of the potential has yet been used here in swtor.
RE: Softcaps discussion - Kaedis - 03-14-2012 12:44 AM
Quote:You certainly have a point about our confidence in drawing conclusions from certain assumptions. While Kor really gets tired of mentioning the ability to draw conclusions from deterministic test ( usually through the floating combat text ), this point isn't valid for mechanics which aren't inherently not deterministic ingame as well.
So your point about the accuracy cap is quite valid. And if we didn't have any ingame character sheet showing us information about crit chance, the whole thing would become extremly uncertain. We still can't validate if the actual crit chance is equal, or in any means correlates to the one displayed in the character sheet. We just assume it does, but we can of course trust this assumption much more than for example the 8% accuracy cap.
But besides that accuracy cap, there is nearly no other non-deterministic problem suspect to inherent distrust. There is no reason to believe the base-damage between min and max values is not evenly distributed, etc.
Ya, Accuracy is a rather sticky point, and one which is a major limitation to current theorycrafting (simulation or otherwise). I tend to forget about it simply because I play a sorcerer, and though we, just as the 8% cap, can't actually prove we don't need the stuff, it's pretty solidly believed that accuracy and sorcs just don't really care for each other. You make a good point about crit, though (particularly since bosses in WoW had crit suppression, though they were higher level compared to the player, a trait not shared in SWTOR).
And yes, I am becoming extremely weary of the community assumption that theorycrafting is woefully inaccurate or pure guesswork without combat logs or damage meters.
RE: Softcaps discussion - Tesuji - 03-14-2012 01:24 AM
(03-13-2012 04:23 PM)Kor Wrote: On a final note, your belief in the failure of other simulators to properly model their game is rather poor reasoning for a blanket vote of no-confidence in the work we've done here on ours.
Actually, no it's not. Many of the same people are using many of the same techniques to solve very similar problems. It's certainly not conclusive, but it absolutely is an important piece of data that any reasonable and objective person would use in evaluating what the likely future success of simulations in TOR will be.
RE: Softcaps discussion - Rouncer - 03-14-2012 01:50 AM
I used to do a lot of work on EnhSim. Not as a coder (can't code) but more as a tester and that is what seems to be missing from this discussion. I think that what is available now is nothing short of amazing for the paucity of data that can be pulled from the game but accuracy is just the tip of the confidence iceberg for me.
Right now so much is reliant on Bioware giving us proper information in those tooltips when that may not be the case. Whether intentional or not there were many small discrepancies between tooltip/formula expectations and what ended up being seen in the combatlog when working through things for EnhSim that I would be surprised if it wasn't similar for any simulator/spreadsheet for SWTOR.
RE: Softcaps discussion - Tesuji - 03-14-2012 02:02 AM
(03-13-2012 05:49 PM)Caltiom Wrote: The old discussion about simulators and combat logs
You certainly have a point about our confidence in drawing conclusions from certain assumptions. While Kor really gets tired of mentioning the ability to draw conclusions from deterministic test ( usually through the floating combat text ), this point isn't valid for mechanics which aren't inherently not deterministic ingame as well.
They need to be deterministic, but more importantly, they need to be known.
Example: for the first 2 months of the game, the Sentinel Watchman ability "Searing Saber" was not working as the tooltip stated. Rather than giving a 15% increase to the critical damage of burn effects, as stated, it instead gave a 15% increase to all burn damage-- an enormous difference.
Now, in the first place, to discover this bug you need to use parses, which is what the person who reported it did. You have no way of discovering this sort of thing using simulation software.
But more importantly, even if this issue were to remain undiscovered, combat parses will take the actual behavior into account. Your simulation may show that it makes sense to drop Plasma Blades, but the combat parse won't. Your simulator may show that it's better to stack Surge in order to maximize the benefit of Plasma Blades, but the combat parse will correctly show that it's better to stack Power.
At any given time, there is close to a 100% chance that at least one class in the game has behavior that significantly differs from (1) how the developers intended it, and/or (2) how the community understands it. Given that 1.2 is going to introduce significant changes skill trees for most/all classes, I'm pretty comfortable saying that in the near future, there's quite likely to be multiple such issues.
Quote:
So your point about the accuracy cap is quite valid.
I don't believe it's just the accuracy cap. Unless I have missed an authoritative post on the subject (which I concede is possible), I don't believe that we have 100% confidence that we understand how excess accuracy is used to reduce the defense of bosses. Given that it's pretty trivial for my Sentinel to hit 113% accuracy without even trying, this is pretty important (and obviously for Sentinels the question is more relevent than for every other class, since we clearly benefit from the offhand accuracy boost).
Quote:But besides that accuracy cap, there is nearly no other non-deterministic problem suspect to inherent distrust.
The combat formulas and values may be deterministic, but the gameplay is not.
Example: during some fights, I will save Transcendence indefinitely, using it as an "oh shit" button for extra defense or extra speed for the group, because I know from prior experience that if things go smoothly we will down the boss without difficulty and don't need my extra DPS from Zen. This is a judgment call based on how the particular group is doing so far that night, whether the same group has run together previously, etc.
Now, it is possible to simulate this without difficulty. But your standard pre-packaged scenarios don't. And there are literally DOZENS of variations from your standard scenarios that occur in almost every fight. I am constantly deviating from the "optimal" DPS rotation in ways that have nothing to do with "delay of x seconds in between casts to account for movement during the fight" type variables.
Quote:Unfortunately there is never that much code review, and we invite everyone here to do so to improve the simulation model. While there are a lot of ways to track the output and potential errors/problem of a ability in SimC, it would often have been easier if the code had been reviewed and human errors had been spotted earlier.
Sure, I'll take a look.
To summarize my position on simulators:
Without an established pattern of verifying that the results of the simulator match against actual combat logs, I do not trust them, even a little. I place more faith in no numbers at all, simply going by gut feel of what seems to work in raids, than in a simulator that's based purely on tooltips and published combat formulas.
With an established track record of verifying that they seem to give reasonable results, I think simulators are a useful tool for highly skilled players who are extremely knowledgable about the simulator to use as an exploratory tool in their quest to understand game behavior.
In all cases, I think statements like "Comparing the data among these 4 types of reports can give you all the information you need for basically any boss mechanic out there" show a faith in the value of simulation output that borders on hubris.
RE: Softcaps discussion - Kaedis - 03-14-2012 02:53 AM
Quote:Now, in the first place, to discover this bug you need to use parses, which is what the person who reported it did. You have no way of discovering this sort of thing using simulation software.
This is precisely my point: we discovered it without parses. FCT is plenty sufficient for finding most of these quirks and oddities. Yes, log parsing makes this significantly easier, particularly to demonstrate, but they are by no means required, as you seem to believe.
Quote:Without an established pattern of verifying that the results of the simulator match against actual combat logs, I do not trust them, even a little. I place more faith in no numbers at all, simply going by gut feel of what seems to work in raids, than in a simulator that's based purely on tooltips and published combat formulas.
Apparently you haven't been paying attention to how we do things around here. Published formulae? Where do you suspect they were published? We derived them, using extensive and tireless testing and verification. These numbers match perfectly with what is in the game files, is displayed on the tooltips (excluding the skill ones, which are known to be inaccurate), and match the FCT and HP bar changes in-game. Yes, all of these could be wrong, but they would all need to be wrong is precise proportion, so it makes no functional difference.
Beyond that, you're making a binary fallacy here. Just because the simulators may not be 100% accurate does not mean that their results are random or useless. We have put a great deal of testing and verification into checking these mechanics. Even if we don't have them down perfectly yet, we've got them damn close, and that means that simulator is still significantly more likely to give the right answer than your "gut feeling".
We don't exactly just blindly trust everything here. We test the shit out of these mechanics, and are constantly reviewing and rechecking our findings. This is science, not guesswork.
RE: Softcaps discussion - Alratan - 03-14-2012 03:23 AM
As this thread seems to have gone away from soft-caps, I think it's time for a lock. Thanks for the stimulating discussions, everyone, but I'm sure we'll see what's what on this topic in the coming weeks.