2 Framework

This paper …

2.1 Types of Questions to Ask

• Differences in documentation
• Differences in modeling assumptions
• Differences in default values
• Differences in available parameters
• Edge cases
• Can default call in one language be replicated in another
  • If yes, do we have numerical equivalence within an acceptable level
• Can a common/historical specification in one language be replicated in another
  • If yes, do we have numerical equivalence within an acceptable level
  • If no, is the common/historical specification the statistically preferred method or should it be abandoned?
• Can the preferred specification in one language be replicated in another?
  • If yes, do we have numerical equivalence within an acceptable level
• Differences in available output statistics
• Differences ‘deep’ in the implementation which are the result of decisions developers make when implementing a particular algorithm
  • May not be clear in the documentation at the level exposed to the user

2.2 Possible Solutions

• Select a method for the analysis plan – be transparent on known alternatives and known differences
• Are there multiple solutions within a given language to consider?