%3CLINGO-SUB%20id%3D%22lingo-sub-1469422%22%20slang%3D%22en-US%22%3ECreating%20high-quality%20exams%3A%20The%20path%20from%20beta%20to%20live%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1469422%22%20slang%3D%22en-US%22%3E%3CP%3EI%E2%80%99ve%20heard%20from%20many%20of%20you%20who%20wonder%20why%20so%20much%20time%20passes%20between%20taking%20a%20beta%20exam%20and%20getting%20your%20results.%20We%20really%20appreciate%20your%20help%20with%20our%20beta%20program.%20I%E2%80%99d%20like%20to%20give%20you%20some%20insight%20into%20the%20beta%20timeline%20for%20creating%20high-quality%20exams%2C%20so%20you%20can%20understand%20why%20it%20might%20take%20as%20few%20as%208%20or%20as%20many%20as%2016%20weeks%20to%20get%20your%20results.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EFinding%20250%20qualified%20candidates%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EWe%20conduct%20beta%20exams%20as%20part%20of%20the%20development%20process%20for%20releasing%20exams.%20Beta%20testing%20lets%20us%20learn%20how%20each%20question%20(aka%20%E2%80%98item%E2%80%99%20in%20my%20world)%20performs%20in%20exam%20conditions.%20To%20get%20the%20best%20information%20about%20the%20psychometric%20performance%20of%20the%20items%20as%20well%20as%20detailed%20verbatim%20feedback%2C%20we%20need%20at%20least%20250%20qualified%20people%20to%20take%20the%20beta%20exam.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20offer%20the%20beta%20exam%20at%20a%20discounted%20price%20for%20a%20four-week%20period.%20If%20fewer%20than%20250%20qualified%20candidates%20sign%20up%2C%20we%20extend%20the%20offer%20for%20a%20few%20more%20weeks%20until%20we%20meet%20our%20threshold%20of%20250%20candidates.%20This%20pushes%20out%20the%20time%20it%20takes%20before%20we%20can%20move%20to%20the%20next%20stage%20of%20exam%20development.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20initial%20process%20takes%20from%20four%20to%20eight%20weeks.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EAnalyzing%20and%20reviewing%20test%20results%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EAs%20soon%20as%20at%20least%20250%20qualified%20candidates%20have%20taken%20the%20test%2C%20we%20pull%20the%20beta%20data%20and%20perform%20a%20detailed%20analysis.%20This%20includes%20reviewing%20a%20wide%20variety%20of%20statistics%20that%20help%20us%20understand%20how%20each%20item%20is%20performing%20and%20how%20various%20combinations%20of%20items%20(called%20%3CEM%3Eforms%3C%2FEM%3E)%20are%20performing.%20We%20also%20review%20time%20spent%20on%20each%20item%20and%20each%20form%2C%20along%20with%20your%20comments.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20use%20this%20information%20to%20determine%20which%20items%20we%E2%80%99ll%20keep%20on%20the%20exam%2C%20which%20items%20we%E2%80%99ll%20remove%2C%20and%20which%20items%20need%20further%20discussion%20with%20subject%20matter%20experts%20(SMEs).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESMEs%20help%20me%20finalize%20the%20test%20item%20pool%20by%20identifying%20items%20that%20can%20be%20fixed%20and%20the%20items%20that%20must%20be%20removed.%20They%20work%20with%20me%20to%20set%20the%20%3CEM%3Ecut%3C%2FEM%3E%20(or%20passing)%20score.%20I%20consolidate%20this%20feedback%20to%20finalize%20the%20item%20pool%20and%20cut%20score.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20takes%20about%20two%20weeks.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EAssembling%20the%20tests%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ENext%2C%20we%20design%20the%20forms%20assembly%20specifications%20(the%20combination%20of%20items%20that%20you%20see%20when%20you%20sit%20for%20the%20exam).%20We%20want%20to%20ensure%20that%20every%20exam%20delivery%20is%20psychometrically%20equivalent%2C%20while%20balancing%20multiple%20other%20considerations.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20takes%20about%20two%20weeks.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EBuilding%2C%20testing%2C%20and%20publishing%20the%20tests%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EAfter%20we%20complete%20the%20forms%20assembly%20specifications%20for%20the%20live%20version%20of%20the%20exam%2C%20we%E2%80%99re%20ready%20to%20build%2C%20test%2C%20and%20publish%20the%20exam.%20We%20work%20with%20one%20of%20our%20exam%20delivery%20partners%20to%20build%20the%20exam%20%E2%80%9Cpackage.%E2%80%9D%20This%20also%20entails%20setting%20up%20the%20exam%20in%20their%20system%20and%20testing%20it%20to%20ensure%20that%20everything%20(including%20scoring)%20is%20functioning%20as%20expected.%20My%20team%20also%20reviews%20this%20pre-published%20version.%26nbsp%3BThen%20the%20exam%20is%20published%20by%20the%20exam%20delivery%20provider%20into%20their%20channel%20so%20that%20you%20can%20take%20the%20exam%20either%20at%20a%20test%20center%20or%20through%20online%20proctoring.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20takes%20about%20two%20weeks.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ERescoring%20beta%20tests%20after%20the%20exam%20goes%20live%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EDuring%20the%20week%20after%20the%20exam%20is%20live%2C%20we%20rescore%20all%20the%20beta%20tests.%20(We%20refer%20to%20this%20as%20%E2%80%9Crescore%E2%80%9D%20because%20each%20beta%20candidate%20is%20rescored%20on%20ONLY%20the%20items%20we%20keep%20on%20the%20exam%20or%20that%20didn%E2%80%99t%20undergo%20significant%20changes%20during%20the%20process%20described%20above.)%20We%20validate%20the%20rescore%20exam%20specs%2C%20review%20them%20for%20accuracy%2C%20and%26nbsp%3Bthen%20apply%20the%20specs%20to%20each%20beta%20delivery.%20We%20conduct%20multiple%20quality%20reviews%20during%20this%20process.%20Every%20beta%20candidate%E2%80%99s%20test%20is%20compared%20to%20the%20list%20of%20items%20that%20were%20kept%20in%20the%20pool%2C%20and%20the%20cut%20score%20applied%20to%20determine%20pass%2Ffail%20status.%20As%20soon%20as%20the%20steps%20are%20complete%2C%20beta%20candidates%20are%20notified%20of%20their%20results.%20%26nbsp%3B%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20takes%20about%20a%20week.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EThanks%20for%20your%20help%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EWe%20hope%20this%20gives%20you%20some%20insight%20into%20our%20beta%20exam%20process%2C%20and%20why%20we%20take%20such%20care%20(and%20so%20much%20time)%20to%20get%20it%20right.%20At%20every%20step%2C%20we%20keep%20our%20quality%20bar%20high.%20The%20result%20is%20an%20exam%20that%20the%20industry%20can%20rely%20on.%20We%20appreciate%20your%20role%20in%20helping%20to%20make%20our%20process%20successful%2C%20and%20we%20thank%20you%20for%20your%20feedback%20and%20insight...and%20your%20patience%20in%20getting%20your%20beta%20exam%20result.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1469422%22%20slang%3D%22en-US%22%3E%3CP%3ELearn%20more%20about%20the%20beta%20exam%20process%20and%20the%20path%20to%20launching%20the%20exam%20live.%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22MSC17_global_013.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F201496iE68B3D993764D142%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22MSC17_global_013.png%22%20alt%3D%22MSC17_global_013.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1469422%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ETips%20%26amp%3B%20Tricks%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1505021%22%20slang%3D%22en-US%22%3ERe%3A%20Creating%20high-quality%20exams%3A%20The%20path%20from%20beta%20to%20live%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1505021%22%20slang%3D%22en-US%22%3E%3CP%3EThat%20explains%20the%20time%20taken%20to%20have%20the%20results%20out.%20I%20believe%20it%20may%20differ%20from%20one%20exam%20to%20the%20other%20depending%20on%20the%20time%20taken%20to%20achieve%20minimum%20no%20of%20exam%20takers%20for%20optimum%20analysis%20.%20Thanks%20for%20the%20insight.%26nbsp%3B%3CIMG%20class%3D%22lia-deferred-image%20lia-image-emoji%22%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Fhtml%2Fimages%2Femoticons%2Fstareyes_40x40.gif%22%20alt%3D%22%3Astareyes%3A%22%20title%3D%22%3Astareyes%3A%22%20%2F%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1563341%22%20slang%3D%22en-US%22%3ERe%3A%20Creating%20high-quality%20exams%3A%20The%20path%20from%20beta%20to%20live%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1563341%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F50590%22%20target%3D%22_blank%22%3E%40Liberty%20Munson%3C%2FA%3E%2C%3C%2FP%3E%3CP%3EAny%20tentative%20date%20to%20get%20the%20AZ-303%20and%20AZ-304%20live%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

I’ve heard from many of you who wonder why so much time passes between taking a beta exam and getting your results. We really appreciate your help with our beta program. I’d like to give you some insight into the beta timeline for creating high-quality exams, so you can understand why it might take as few as 8 or as many as 16 weeks to get your results.

 

Finding 250 qualified candidates

We conduct beta exams as part of the development process for releasing exams. Beta testing lets us learn how each question (aka ‘item’ in my world) performs in exam conditions. To get the best information about the psychometric performance of the items as well as detailed verbatim feedback, we need at least 250 qualified people to take the beta exam.

 

We offer the beta exam at a discounted price for a four-week period. If fewer than 250 qualified candidates sign up, we extend the offer for a few more weeks until we meet our threshold of 250 candidates. This pushes out the time it takes before we can move to the next stage of exam development.

 

This initial process takes from four to eight weeks.

 

Analyzing and reviewing test results

As soon as at least 250 qualified candidates have taken the test, we pull the beta data and perform a detailed analysis. This includes reviewing a wide variety of statistics that help us understand how each item is performing and how various combinations of items (called forms) are performing. We also review time spent on each item and each form, along with your comments.

 

I use this information to determine which items we’ll keep on the exam, which items we’ll remove, and which items need further discussion with subject matter experts (SMEs).

 

SMEs help me finalize the test item pool by identifying items that can be fixed and the items that must be removed. They work with me to set the cut (or passing) score. I consolidate this feedback to finalize the item pool and cut score.

 

This takes about two weeks.

 

Assembling the tests

Next, we design the forms assembly specifications (the combination of items that you see when you sit for the exam). We want to ensure that every exam delivery is psychometrically equivalent, while balancing multiple other considerations.

 

This takes about two weeks.

 

Building, testing, and publishing the tests

After we complete the forms assembly specifications for the live version of the exam, we’re ready to build, test, and publish the exam. We work with one of our exam delivery partners to build the exam “package.” This also entails setting up the exam in their system and testing it to ensure that everything (including scoring) is functioning as expected. My team also reviews this pre-published version. Then the exam is published by the exam delivery provider into their channel so that you can take the exam either at a test center or through online proctoring.

 

This takes about two weeks.

 

Rescoring beta tests after the exam goes live

During the week after the exam is live, we rescore all the beta tests. (We refer to this as “rescore” because each beta candidate is rescored on ONLY the items we keep on the exam or that didn’t undergo significant changes during the process described above.) We validate the rescore exam specs, review them for accuracy, and then apply the specs to each beta delivery. We conduct multiple quality reviews during this process. Every beta candidate’s test is compared to the list of items that were kept in the pool, and the cut score applied to determine pass/fail status. As soon as the steps are complete, beta candidates are notified of their results.   

 

This takes about a week.

 

Thanks for your help

We hope this gives you some insight into our beta exam process, and why we take such care (and so much time) to get it right. At every step, we keep our quality bar high. The result is an exam that the industry can rely on. We appreciate your role in helping to make our process successful, and we thank you for your feedback and insight...and your patience in getting your beta exam result.

2 Comments
Senior Member

That explains the time taken to have the results out. I believe it may differ from one exam to the other depending on the time taken to achieve minimum no of exam takers for optimum analysis . Thanks for the insight. :stareyes:

Senior Member

Hi @Liberty Munson,

Any tentative date to get the AZ-303 and AZ-304 live?