Loading
Created by:fauxlosopherfauxlosopher
{"id":760433,"type":0,"block_id":"fauxlosopher_1363307327","theme_id":31,"user_id":261822,"path":"GuideToCalibration","title":"The Analyst&#39;s Guide to Calibration","pubtitle":"The Analyst&#39;s Guide to Calibration","tags":"","public":true,"publicAccess":true,"private_link_enabled":0,"thumb":"https://s3-eu-west-1.amazonaws.com/infogram-thumbs-200/fauxlosopher_1363307327.jpg","width":700,"copyright":null,"properties":{"exportsettings":"{\"CHART_1142253\":[],\"CHART_1142378\":[],\"CHART_1141726\":[],\"CHART_1141826\":[],\"CHART_1142153\":[],\"CHART_1142523\":[]}","transparent":false,"rtl":false,"export_settings":{"showGrid":true,"showValues":true},"whitelabel":false,"embed_button":"enabled","title_link":"infogram","custom_logo":"infogram","custom_link_url":"","embed_button_text":"Share","decimal_separator":".,"},"elements":[{"type":"particle","particle_id":14349745,"object_id":"f54f7f28-ea63-11e4-a037-7b5f270fdca9","particle_type":"maintitle","picture":null,"text":"The Analyst&#39;s Guide to Calibration","content_type":"","title":null,"shrink":null},{"type":"particle","particle_id":732257,"object_id":"fauxlosopher1363318833","particle_type":"bodytext","picture":null,"text":"The purpose of this guide is to inform analysts of a solution to a commonly experienced dilemma in making predictive judgments. During normal conversation, we often speak of probabilities and levels of confidence as percentages: \"I&#39;m 90% sure of this, 40% certain of that,\" and so on. But for as intuitive as this kind of quantified expression is, we do not use it in our everyday work. Why not? Well, for one thing, it makes us more accountable. Numbers are precise and concrete, and we fear being wrong about an unambiguous expression once it goes on record. But the larger issue is this: there&#39;s no way of knowing that what we say and actually mean are one and the same. Or, even worse, we don&#39;t know that when we communicate to someone else that something has a \"such-and-such percent chance&#39;\" of occurring or being true, that THAT person has the same understanding of this measure as we do! But fear not, faithful analyst. There is a way to solve this problem. It&#39;s called \"calibration.\"","content_type":"","title":"","shrink":null},{"type":"particle","particle_id":732270,"object_id":"fauxlosopher1363316994","particle_type":"bodytitle","picture":null,"text":"This is what it means for us to be \"calibrated\" if we assess, say, a 70% confidence in something:","content_type":"","title":"","shrink":null},{"type":"chart","chart_id":1142253,"object_id":"fauxlosopher_1363315410","chart_type_nr":4,"data":[[["","CALIBRATED"],["What We Say","70%"],["What We Mean","70%"]]],"custom":{"xlabel":"Confidence ( % )","height":"250","absoluteDistribution":false,"legacyTransposeDone":true,"switchRowsColumns":true},"colors":["#339bd6","#ff414d","#5bb7c3","#959595","#287f8a","#4c4c4c","#005863","#000000","#9af3ff","#e2e2e2"],"dimensions":[],"modifier":1,"sheetnames":[],"sheets_settings":[]},{"type":"particle","particle_id":732280,"object_id":"fauxlosopher1363317341","particle_type":"bodytitle","picture":null,"text":"...And this is what it looks like when we are NOT calibrated:","content_type":"","title":"","shrink":null},{"type":"chart","chart_id":1142378,"object_id":"fauxlosopher_1363317144","chart_type_nr":4,"data":[[["","NOT CALIBRATED"],["What We Say","70%"],["What We Mean","50%"]]],"custom":{"xlabel":"Confidence ( % )","height":"250","legacyTransposeDone":true,"switchRowsColumns":true},"colors":["#339bd6","#ff414d","#5bb7c3","#959595","#287f8a","#4c4c4c","#005863","#000000","#9af3ff","#e2e2e2"],"dimensions":[],"modifier":1,"sheetnames":[],"sheets_settings":[]},{"type":"particle","particle_id":732295,"object_id":"fauxlosopher1363317585","particle_type":"bodytitle","picture":null,"text":"(In this case, we are \"overconfident\" by 20%. More on that later.)","content_type":"","title":"","shrink":null},{"type":"particle","particle_id":732248,"object_id":"fauxlosopher1363316311","particle_type":"bodytitle","picture":null,"text":"So how can we determine if analysts are calibrated or not? We test them! First, we give the analysts 10 questions to answer and assign confidence levels to, as a percentage. A single set might look like this:","content_type":"","title":"","shrink":null},{"type":"particle","particle_id":731869,"object_id":"fauxlosopher_1363307327","particle_type":"quote","picture":null,"text":"Confidence in judgments for 10 analytic assessments","content_type":"","title":"","shrink":null},{"type":"chart","chart_id":1141726,"object_id":"fauxlosopher_1363307327","chart_type_nr":6,"data":[[["Calibration Questions","1","2","3","4","5","6","7","8","9","10"],["PERCENT CONFIDENCE ASSIGNED TO EACH QUESTION ANSWERED","70%","50%","80%","80%","90%","50%","60%","60%","90%","70%"]]],"custom":{"height":"400","legacyTransposeDone":true,"switchRowsColumns":true},"colors":["#339bd6","#bfbfbf","#5bb7c3","#959595","#287f8a","#4c4c4c","#005863","#000000","#9af3ff","#e2e2e2"],"dimensions":[],"modifier":0,"sheetnames":[],"sheets_settings":[]},{"type":"particle","particle_id":731998,"object_id":"fauxlosopher1363309884","particle_type":"bodytitle","picture":null,"text":"Then we check to see how their STATED confidence lines up with their ACTUAL performance. In the example given, the average stated confidence made across 10 analytic assessments was 70%. We then compare this average confidence with the percent of answers they got right to find out how well they performed, like this:","content_type":"","title":"","shrink":null},{"type":"chart","chart_id":1141826,"object_id":"fauxlosopher_1363308756","chart_type_nr":19,"data":[[["Assessment Questions","Correct","Incorrect"],["1","x"],["2","","x"],["3","","x"],["4","x"],["5","x"],["6","","x"],["7","","x"],["8","x"],["9","","x"],["10","x"]]],"custom":{"height":"425","showRowHeader":true,"showColHeader":true},"colors":["#ff414d"],"dimensions":[],"modifier":0,"sheetnames":[],"sheets_settings":[]},{"type":"particle","particle_id":732028,"object_id":"fauxlosopher1363310214","particle_type":"bodytitle","picture":null,"text":"The analyst in this case only got 50% (5 out of 10) of the questions correct. That means that:","content_type":"","title":"","shrink":null},{"type":"chart","chart_id":1142153,"object_id":"fauxlosopher_1363313443","chart_type_nr":5,"data":[[["title","Calibration Status"],["MIN","-50"],["MAX","50"],["Value","20"],["Label","CALIBRATION STATUS ( 20% overconfident )"]]],"custom":{"height":"250"},"colors":["#ff414d","#bfbfbf","#5bb7c3","#959595","#287f8a","#4c4c4c","#005863","#000000","#9af3ff","#e2e2e2"],"dimensions":[],"modifier":0,"sheetnames":[],"sheets_settings":[]},{"type":"particle","particle_id":732187,"object_id":"fauxlosopher1363315183","particle_type":"bodytitle","picture":null,"text":"The analyst was 20% overconfident!","content_type":"","title":"","shrink":null},{"type":"particle","particle_id":695279,"object_id":"fauxlosopher1363320589","particle_type":"bodytitle","picture":null,"text":"Once calibrated, the same analyst&#39;s calibration status should look like this, being neither over- nor under- confident:","content_type":"","title":"","shrink":null},{"type":"chart","chart_id":1142523,"object_id":"fauxlosopher_1363320115","chart_type_nr":5,"data":[[["title","Calibration Status"],["MIN","-50"],["MAX","50"],["Value","0"],["Label","IDEAL CALIBRATION STATUS ( 0% over- & under- confident )"]]],"custom":{"height":"250"},"colors":["#339bd6","#c5c5c5","#5bb7c3","#959595","#287f8a","#4c4c4c","#005863","#000000","#9af3ff","#e2e2e2"],"dimensions":[],"modifier":0,"sheetnames":[],"sheets_settings":[]},{"type":"particle","particle_id":732391,"object_id":"fauxlosopher1363320753","particle_type":"bodytext","picture":null,"text":"The good news is that over time, and with practice, analysts CAN get better at this -- and can even achieve ideal calibration status, thereby allowing them to QUANTIFY their analytic assessments. It takes hard work and constant self-evaluation of one&#39;s own performance against the ideal, but with each successive set of questions (usually trivia), the analyst WILL get better. Think of analytic calibration as inferential fitness: you have to keep at it, or you&#39;ll lose it. But if you hit the mental gym often and continue to focus on your critical thinking skills, you&#39;ll soon see results. How confident am I? About 90%.","content_type":"","title":"","shrink":null},{"type":"particle","particle_id":732445,"object_id":"fauxlosopher1363322913","particle_type":"quote","picture":null,"text":"Quantify your assessments, Improve your inferential fitness, and gain confidence in your confidence.","content_type":"","title":"","shrink":null},{"type":"particle","particle_id":744119,"object_id":"fauxlosopher1363619492","particle_type":"bodytext","picture":null,"text":"Infographic created by A. Schroeder, 2013.","content_type":"","title":"","shrink":null},{"type":"particle","particle_id":740393,"object_id":"fauxlosopher1363564490","particle_type":"bodytext","picture":null,"text":"Source: Hubbard, Douglas W. \"How To Measure Anything: Finding the Value of Intangibles in Business.\" John Wiley & Sons. Hoboken, NJ: 2007.","content_type":"","title":"","shrink":null}],"theme":{"createdAt":"2016-04-22T04:54:07.000Z","updatedAt":"2016-12-28T10:33:43.000Z","logoImages":[""],"charts":{"treemap":{"labels":{"name":{"fontFamily":"PT Sans Narrow","fontWeight":"700"},"value":{"fontFamily":"PT Sans Narrow"}}},"wordcloud":{"labels":{"fontFamily":"Arial","fontWeight":"700"}},"table":{"cellBackground":"#ffffff","headerBackground":"#eeeeee","cellColor":"#232323","headerColor":"#232323","shapeFill":"#232323"}},"color":{"bg":"#ffffff","text":"#000000","chart":{"bg":"transparent","text":"#767676"},"element":{"bg":"transparent","text":"#000000"}},"colorPresets":[],"localFonts":[],"font":{"common":{"textAlign":"initial","fontSize":"16","fontStyle":"normal","fontWeight":"400","fontFamily":"PT Sans Narrow"},"legend":{"fontSize":"16","fontWeight":"400"},"label":{"fontSize":16,"fontWeight":400}},"fontPresets":[],"fontFamilies":["Pontano Sans","PT Sans","PT Sans Narrow","Arial Narrow"],"id":31,"title":"Light-new","usergroup":"archived","picture":"https://d1m5pq7b4fzvad.cloudfront.net/31/thumbnail.png","thumb":"/i/templates/S/grey.png","order":28,"public":0,"width":550,"fonts":"Pontano+Sans;PT+Sans+Narrow","colors":["#34a6b5","#bfbfbf","#5bb7c3","#959595","#287f8a","#4c4c4c","#005863","#000000","#9af3ff","#e2e2e2"],"logocolor":"6abac5","logoUrl":"","showLogo":"","showEmbed":"","embedButtonText":"Share","top":0,"padding":20,"spacing":10,"shrinkMargin":20,"shrinkPadding":20,"spacingElementMin":null,"spacingElementMax":null,"css":".heatmap-legend {\n\tmargin-bottom: 10px;\n\tbackground: rgba(255, 255, 255, 0.7);\n}\n.igc-textual-figure {\n\tfont-size: 45px;\n}\n.igc-sheets .igc-sheet .igc-sheet-ico,\n.igc-sheets .igc-sheet:hover .igc-sheet-ico {\n\tborder-color: #7d7d7d;\n}\n.igc-sheets .igc-sheet .igc-sheet-label,\n.igc-sheets .igc-sheet:hover .igc-sheet-label,\n.igc-sheets .igc-sheet.active .igc-sheet-label {\n\tcolor: #7d7d7d;\n}\n.igc-sheet .igc-sheet-ico,\n.igc-sheet.active .igc-sheet-ico {\n\tbackground: transparent;\n}\n.igc-sheets .igc-sheet.active .igc-sheet-ico::after,\n.igc-sheets .igc-sheet:hover .igc-sheet-ico::after {\n\theight: 14px;\n\twidth: 14px;\n\tleft: 2px;\n\ttop: 2px;\n\tbackground: #7d7d7d;\n}\ndiv[id^='table_'] .inner {\n\tbackground: transparent;\n\tpadding-left: 0;\n\tpadding-rigth: 0;\n}\n.igc-table-scroll {\n\tbackground: #eee;\n}\n.chart > .inner {\n\tpadding-top: 21px;\n\tpadding-left: 21px;\n\tpadding-right: 21px;\n}\n.chart > .igc-chart.inner\n.ig-copyright,\n#madewith a:link,\n#madewith a:active,\n#madewith a,\n#madewith a:visited,\n#madewith a:hover {\n\tcolor: #7d7d7d !important;\n}\n.ig-container {\n\tbackground: url(/i/templates/31/fons2.png);\n}\n.headline {\n\tfont-family: Pontano Sans;\n\tfont-weight: 400;\n\tfont-size: 40px;\n\tcolor: #34a6b5;\n\ttext-transform: uppercase;\n\ttext-align: center;\n\tline-height: 50px;\n}\n.chart-title {\n\tbackground: rgba(52, 166, 181, 0.7);\n\tfont-family: Pontano Sans;\n\tfont-weight: 400;\n\tfont-size: 22px;\n\tcolor: #fff;\n\ttext-align: center;\n}\n.chart-title .innertext {\n\tpadding: 3px;\n}\n.bodytext {\n\tfont-family: Pontano Sans;\n\tpadding: 20px;\n\tpadding-top: 10px;\n\tpadding-bottom: 10px;\n\tfont-size: 16px;\n\tcolor: #464646;\n}\n.quote {\n\tpadding-top: 10px;\n\tpadding-bottom: 10px;\n\tfont-family: Pontano Sans;\n\ttext-align: center;\n\tfont-weight: 400;\n\tfont-size: 27px;\n\tcolor: #34a6b5;\n\ttext-transform: uppercase;\n\tpadding-left: 30px;\n\tpadding-right: 30px;\n}\n.quotetitle {\n\tcolor: #3e4648;\n\tfont-size: 22px;\n}\n.chart .inner,\n.map {\n\tbackground: #f7f7f7;\n}\n.chart {\n\tpadding-bottom: 10px;\n\theight: auto !important;\n}\n.headline.active:hover {\n\tcolor: #0d6874;\n}\n#madewith.active a:hover {\n\tcolor: #0d6874 !important;\n}\n.titlepopup {\n\ttop: 1px;\n}\n.titlepopup1,\n.titlepopup2 {\n\tborder-color: #0d6874;\n}\n.shrink .chart > .inner {\n\tpadding-left: 10px;\n\tpadding-right: 10px;\n}\n.shrink .chart {\n\tmargin-left: 0;\n\tmargin-right: 0;\n}\n.ig-separator-line {\n\tbackground: rgba(118, 118, 118, 0.5);\n}\n.heatmap-legend-item,\n.heatmap-label {\n\tcolor: #767676;\n\tfont-size: 11px;\n\tfont-family: PT Sans;\n}\n.igc-graph-pie-piece {\n\tstroke: rgba(255, 255, 255, 0.7);\n}\n.igc-tabs .igc-tab-active {\n background: #ffffff;\n }\n .igc-tabs .igc-tab .igc-tab-content,\n .igc-tabs .igc-tab.icon-down:after {\n font-family: PT Sans Narrow;\n color: #000000;\n font-weight: 400;\n }\n .igc-tabs .igc-tab,\n .igc-tab-switcher {\n color: #000000;\n } .igc-tabs.igc-tabs-dropdown .igc-tab-name {\n font-family: PT Sans Narrow;\n color: #000000;\n font-weight: 400;\n } .captiontext {font-family: PT Sans Narrow;color: #000000;font-weight: 400;} .captiontext .innertext {line-height: 1.5;}","chartOptions":{"waterfall":{"colors":"005863 34a6b5 959595"},"candle":{"colors":"959595 34a6b5"},"map":{"showLegend":false,"countryyStroke":"#e1e1e1","countryFill":"#e1e1e1","canvasFill":"transparent"},"common":{"textStyle":{"font-family":"PT Sans Narrow, Arial Narrow","font-size":16,"cursor":"default","fill":"#5a5a5a"},"backgroundStyle":{"stroke-opacity":0,"stroke-width":0},"_gridOptions":{"fontStyle":{"font-family":"PT Sans","font-size":11,"text-anchor":"start","fill":"#767676","cursor":"default"},"lineStyle":{"stroke-width":1,"stroke":"#767676","fill":"none","stroke-dasharray":"."}},"legendOptions":{"font-size":16,"font-family":"PT Sans Narrow"},"tooltipOptions":{"nameFontStyle":{"font-family":"PT Sans Narrow","text-anchor":"start","font-weight":700,"font-size":16,"fill":"#ffffff"},"valueFontStyle":{"font-family":"PT Sans Narrow","text-anchor":"start","font-size":16,"fill":"#fff"},"containerStyle":{"stroke":"#464646","fill":"#464646"}}}},"version":1,"archived":true,"tabs":false,"zeroPaddingEmbed":true,"created_at":"2016-04-22T04:54:07.000Z","updated_at":"2016-12-28T10:33:43.000Z"},"embed":"<script id=\"infogram_0_GuideToCalibration\" title=\"The Analyst&amp;#39;s Guide to Calibration\" src=\"//e.infogr.am/js/dist/embed.js?sQS\" type=\"text/javascript\"></script><div style=\"padding:8px 0;font-family:Arial!important;font-size:13px!important;line-height:15px!important;text-align:center;border-top:1px solid #dadada;margin:0 30px\"><a href=\"https://infogr.am/GuideToCalibration\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">The Analyst&#39;s Guide to Calibration</a><br><a href=\"https://infogr.am\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">Create your own infographics</a></div>","embedIframe":"<iframe src=\"//e.infogr.am/GuideToCalibration?src=embed\" title=\"The Analyst&amp;#39;s Guide to Calibration\" width=\"700\" height=\"\" scrolling=\"no\" frameborder=\"0\" style=\"border:none;\"></iframe><div style=\"padding:8px 0;font-family:Arial!important;font-size:13px!important;line-height:15px!important;text-align:center;border-top:1px solid #dadada;margin:0 30px\"><a href=\"https://infogr.am/GuideToCalibration\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">The Analyst&#39;s Guide to Calibration</a><br><a href=\"https://infogr.am\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">Create your own infographics</a></div>","embedWordpress":"[infogram id=\"GuideToCalibration\" prefix=\"RMU\" format=\"interactive\" title=\"The Analyst&amp;#39;s Guide to Calibration\"]","embedWordpressImage":"[infogram id=\"GuideToCalibration\" prefix=\"8Kg\" format=\"image\" title=\"The Analyst&amp;#39;s Guide to Calibration\"]","embedAsync":"<div class=\"infogram-embed\" data-id=\"GuideToCalibration\" data-type=\"interactive\" data-title=\"The Analyst&#39;s Guide to Calibration\"></div><script>!function(e,t,n,s){var i=\"InfogramEmbeds\",o=e.getElementsByTagName(t),d=o[0],a=/^http:/.test(e.location)?\"http:\":\"https:\";if(/^\\/{2}/.test(s)&&(s=a+s),window[i]&&window[i].initialized)window[i].process&&window[i].process();else if(!e.getElementById(n)){var r=e.createElement(t);r.async=1,r.id=n,r.src=s,d.parentNode.insertBefore(r,d)}}(document,\"script\",\"infogram-async\",\"//e.infogr.am/js/dist/embed-loader-min.js\");</script><div style=\"padding:8px 0;font-family:Arial!important;font-size:13px!important;line-height:15px!important;text-align:center;border-top:1px solid #dadada;margin:0 30px\"><a href=\"https://infogr.am/GuideToCalibration\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">The Analyst&#39;s Guide to Calibration</a><br><a href=\"https://infogr.am\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">Create your own infographics</a></div>","embedImageAsync":"<div class=\"infogram-embed\" data-id=\"GuideToCalibration\" data-type=\"image\" data-title=\"The Analyst&#39;s Guide to Calibration\"></div><script>!function(e,t,n,s){var i=\"InfogramEmbeds\",o=e.getElementsByTagName(t),d=o[0],a=/^http:/.test(e.location)?\"http:\":\"https:\";if(/^\\/{2}/.test(s)&&(s=a+s),window[i]&&window[i].initialized)window[i].process&&window[i].process();else if(!e.getElementById(n)){var r=e.createElement(t);r.async=1,r.id=n,r.src=s,d.parentNode.insertBefore(r,d)}}(document,\"script\",\"infogram-async\",\"//e.infogr.am/js/dist/embed-loader-min.js\");</script><div style=\"padding:8px 0;font-family:Arial!important;font-size:13px!important;line-height:15px!important;text-align:center;border-top:1px solid #dadada;margin:0 30px\"><a href=\"https://infogr.am/GuideToCalibration\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">The Analyst&#39;s Guide to Calibration</a><br><a href=\"https://infogr.am\" style=\"color:#989898!important;text-decoration:none!important;\" target=\"_blank\" rel=\"nofollow\">Create your own infographics</a></div>"}