ࡱ> `!:4uڄ)S !k(xTilTU}Ldf:@7N鴳.qH**" @*Ŋ`LJ+y14$<JK!3Ec?9, HF'BAB"G6N HOF_ZÞ b' N y,Æ5Bh/2e㤗 By,]!Tq\OnJ:|,8)g{p5I9B2[ݣ~'$f 1F8 Xx~Π-0ȓKæ:"ȻIe)z4Rg^.w2urrY"p:rtL6I5lO5\g~gUK<1o-a$7~W Ve3^D#xKF)̧xy'ו.AyjZ|&ȗ}~ӯdlZڅ:R$Je:lʼnO&YH5(LZ} V|q|X8גGY:+`Fs}?+cz$OC-  X#Cc$?'?%`oC4 %PN!q>h?ENIFa^*8B]Ifry5S%ፅOF™cƌQٕ" vB2b!ZO>GKr!NǠAy{i-V-쬷Te7K;"u+,+ϖ|Bi4;Lg4Z K:%8%yT XO@_~Y&rtx7*.б*7 #'JE ll ~ǥcɕ ;50蠽^m/`&9::ʮU6/]ˊ= O7sKpG}~SKe-#W:J5.GE UZNUJ8tP1hqCõ6 OװMi*kkdm(Ϭ`3h/Yn1"vg.+3m|מ 5Xra9#Nפi08z^vV6l^dk"UmȧpΦ6⃛N9t4t`% st`-) m%͡6rcy, p;, WBywOHG9` s,s qT8k`>˹F+I3VQ,>QzBzG7d?gSx*oþ9SbEtQ[6%$?F( -f/ 0DTimes New Roman0Wo 0DArialNew Roman0Wo 0c .  @n?" dd@  @@``  |^     &LO  ,2$:4u AA1?@8Z[h ʚ;ԉr8ʚ;g4DdDd0ppp@ <4!d!d 0L<4dddd 0L D<4BdBd. 0L080___PPT10 ?p-27-Jul-2005 >Study of Qualification CriteriaO ==?STUDY OF QUALIFICATION CRITERIA FOR SOFTWARE VERIFICATION TOOLS@@ A report presented at the 2005 National Software & Complex Electronic Hardware Standardization Conference 27-July-2005 Vdot Santhanam PRINCIPAL INVESTIGATOR BOEING >Pj.TOPICSOverview of Project Phase 1 Summary Phase 2 Summary Phase 3 Activities Test Suite Framework Model Test Suite Results Summary Recommendations for Future Work <G6!G6!Overview of ProjectBObjective of Study Investigate qualification criteria for software structural coverage verification tools Determine whether the regulatory guidance provides sufficient basis for determining whether an automated verification tool enforces the DO-178B coverage criteria accurately Recommend means for improving the objectivity and uniformity of tool qualification process Study organized into three phases Phase 1  Research the issues Phase 2  Study and recommend means to address the issues Phase 3  Demonstrate the efficacy of recommendationsTZ_Z"ZZ_"Phase 1 FindingsPhase 1 of the project found that The current regulatory guidance to be the source of many ambiguities Ambiguities allowed tool vendors and regulatory authorities to interpret coverage criteria in varied ways The study surveyed twenty-one tools from nineteen vendors Most offered coverage analysis per DO-178B levels A, B, and C The basis for an objective set of qualification criteria should begin by clarifying the DO-178B intent The feasibility of developing a test suite to improve objectivity should be investigatedV"x"xPhase 2 Findings)Phase 1 identified issues dealing with the interpretation of DO-178B structural coverage criteria Statement Coverage Decision Coverage MC/DC Phase 2 studied ways to resolve these issues and made recommendations Phase 2 also found that a test suite to bring about uniform interpretation is feasibleBbZ+ZZb+Statement Coverage IssuesStatement Coverage Issues Should implicit statements be subject to coverage? Recommendation: No. Should declarative statements be subject to coverage? Recommendation: Yes, if the declaration generates executable object code.b36J36J  !Decision Coverage Issues (1 of 2)Decision Coverage Issues What is a Decision? Recommendation: Binary valued expressions that are: (a) declared as Boolean, or (b) interpreted as Boolean in one or more contexts, or (c) derive their values from other such expressions How are Boolean constants to be treated? Recommendation: Not subject to coverage How are exception handlers to be treated with respect to entry and exit coverage? Recommendation: Each handler should be subject entry/exit coverage and statements in it subject to decision coverage)(Ru )(R  uXM!Decision Coverage Issues (2 of 2)Decision Coverage Issues What are the contexts in which Boolean expressions should be subject to decision coverage? Recommendations: Should only apply to those that appear in branching constructs Should be renamed as Branch Coverage The definition of a decision as any Boolean expression should be retained for MC/DC purposes This is in contrast to recommendations of CAST-10 Position Paper Presented the new recommendations and rationale at the CAST meeting, Seattle, July 2004.[[U_ MC/DC Issues (1 of 3)Modified Condition/Decision Coverage Issues How should decisions containing short-circuit operators be treated Recommendation: Treat each short-circuited term as an independent, top-level decision, in harmony with the flow graph model suggested by DO-248B The study also noted that this does not mean that branch coverage for Boolean expressions containing arbitrary set of short-circuit operators is equivalent to MC/DC<,C6,C6MC/DC Issues (2 of 3)What are conditions, decisions, Boolean operators? Recommended definitions: Condition: A lowest-level Boolean expression that is: (a) A Boolean variable (including array element and record component), OR (b) a Boolean function call, OR (c) an expression consisting of non-Boolean terms and predefined operators, delivering a Boolean result Boolean operator: An operator that operates on one or more Boolean operands and delivers a Boolean result Decision: (a) A condition, OR (b) the result of a Boolean operator 3ZZZ    Y :YNMC/DC Issues (3 of 3)How is the apparent contradiction in MC/DC as it applies to decisions containing coupled (replicated) conditions to be resolved? DO-178B states that each occurrence of a condition must be treated as a separate condition However, to show independence, each condition must be toggled while holding all other conditions fixed Tackling this issue was the single greatest challenge for the study[hD % hDResolving MC/DC ContradictionbPhase 2 investigated eight variants of MC/DC Five that are different interpretations of the DO-178B definition of MC/DC These variants are referred to here as  flavors Three that could be considered alternate forms of Boolean expression structural coverage These variants are referred to as  alternates b-L1Y/-L1Y/ &Resolving MC/DC Contradiction (contd.)"' @The study concluded Among the five flavors of DO-178B definition [UCM] Unique-Cause MC/DC is the simplest, but not applicable to decisions containing coupled conditions [MSM] Masking MC/DC is the most widely applicable and the most complex [CCM] Coupled-Cause MC/DC is as widely applicable as MSM, but is weaker than MSM:--3+&Resolving MC/DC Contradiction (contd.)"' #The study also concluded Among the three alternate forms of Boolean expression coverage [OCC] Operator Coverage Criterion most closely matches the intent of DO-178B and is significantly simpler to describe and implement OCC is weaker than MSM, but does that matter? More research is needed..@??4,MC/DC Issues (3 of 3, contd.)" The study recommended For the near-term, accept MSM or CCM as meeting the DO-178B requirement For DO-178C, study alternate forms of structural coverage to replace MC/DC Recommend OCC as a starting point@""6-Phase 3 ActivitiesPhase 3 activities consisted of Formulating a test suite Defining test objectives Constructing a prototype test suite Running the tests against selected tools Making recommendations on the development of a full-scale test suite&  8/Test Suite ObjectivesPrimary Objectives for a Tool Qualification Test Suite Should be applicable to tools for popular languages, with ability to exercise language-specific constructs Should be applicable to tools used to verify application software at level A, B, or C Should be tailorable to multiple tools, and multiple compilers (if tool is compiler-independent) Should minimize manual activity required to run the test suite against a given tool*7v7v90Test Suite Formulation@Boeing has constructed a framework for a test suite, called CATS-178B, that meets the primary objectives Addresses coverage at all three levels of criticality Is largely language independent, with ability to include language-specific tests Is largely tool independent, with tailorable scripts to invoke tool and compiler&ii)&Test Suite Formulation (contd.)  CATS is organized by criticality level of the software to be verified by the tool Level C: Statement Coverage Level B: Decision or Branch Coverage Level A: (Flavors of) MC/DC A tool needing qualification at a given level must pass all tests applicable to the lower level(s):R]cR]c7.Test Suite Formulation (contd.)  At each level CATS includes affirmative and negative tests Affirmative tests Confirm the tool correctly reports coverage as attained from a given set of test cases Negative tests Confirm the tool correctly reports coverage as deficient from a given set of test cases Discriminating tests Determine the specific interpretation used by the tool from among acceptable alternate interpretations Failure of a tool on a negative test is more serious than a failure on an affirmative test;PPWPPXPPgP[P;/ /    g  [  *'Test Suite Formulation (contd.)  Tests in CATS are based on test objectives that vary by the level of criticality for which the tool is to be qualified the degree of sophistication of the tests Basic tests: expect all tools to pass without difficulty or variation in results Advanced tests: expect a tool to pass unless the tool s limitations are clearly documented and the application will not violate them@8ZjZZ8j+(Test Suite PrototypeCATS/p Uses a Test Description Language specifically designed to allow generic test descriptions Includes a cross-section of tests from all levels 27 Level C, 23 Level B, 115 Level A Includes Affirmative, Negative or Discriminator tests Includes an Indeterminate class for tests whose validity will depend on of resolution of coverage issues identified in phase 2 Includes tests to exercise Ada-unique constructs Includes customizable scripts to generate drivers and invoke the tool on themZZ$Z6ZZZ$6,)Phase 3 ResultsoFeasibility of a Test Suite that could be used to improve objectivity and uniformity in tool qualification was demonstrated A test suite framework was formulated to meet all major objectives A model test suite was written based on that framework to demonstrate the feasibility The model was validated against three tools Referred to as Tools A, B, and C for anonymity:|/|/J@Tool A ResultsVKTool B ResultsWLTool C ResultsLBCATS/p Preliminary ResultsiSummary of preliminary results The test suite was effective in finding significant deficiencies in all three tools Tests were found to be portable across tools Output reports had to be manually reviewed due to lack of uniform presentation We believe that vendors will address the deficiencies if validation were required, leading to more uniform interpretation&KKRHRecommendations for Future Work The development of a full-scale test suite should be launched The use of a standard test suite to validate coverage analysis will bring objectivity and uniformity to the tool qualification process The test suite could serve as a  final authority in resolving ambiguities in any natural language statement of the requirements&>>; ` 3` 999MMM` f` f3` 3>?" dd@x? " }d@i dA"  n?" dd@   @@``P\     `<p>> 77lf7(  l6 l Tgֳgֳ ?"|K  FClick to edit Master text styles Second level Third level Fourth level!    G l Zgֳgֳ ?"0  T Click to edit Master title style! ! l  `g5%g5% ?"c0  L Page *  l  `gg ?"t    D*  l  `Hgֳgֳ ?"``  B* 1T  l " l BCDE F kk` n p{ Q 2      n L +     { R *     e >   ~ p _ dO A= *   bD(r W;xc@y-2 K2  = \h |7  u#FGja4fB`:k,t: '^E+gqG, Y"|YcM9'@~:{  *Fb~ '/7?HR9\Sgmr}  / F [ r  ! 3   B p # N w y r l h: d^ b a b d h l r7 zQ j  0 Kf! C e      9m ]S 8     - Nv nQ +      s G0 R u    n P 3       j P 6            w g U C 0        x _ D (    s , T     / ^' u/ 6 < C I N S W 8[ Q_ kb d f g h h h +f Ld kb ^ Y T O H $A A8 _0 |'      $ ? Z t    q _ L 9 4% K a P H ? # 7 0 2 !             a   A Y  } o b b Q ? a -  _   ` _S B. &  _2zvu]qFm-ieca`__{`?emwX!P&FhlC2`z`&J\5":t /Qr 1Q'o2<IUco49oUNT#0imBM0{iW3G~z _S 1       ( 8 kH SW ;g "u       i M 1       e G (       X )      o D   } f uM M3 ( > S k O  w?:^W!Lex&k6,il,>wj*&b -$F<_Uzl&Ff  # 1 < H /Q RZ va i n r u /w Uw {w v s p l g >a ]Z }S K A 8 - ! 1 M jXF3lF  lJi(O4@`@`@`  l Bi CDE4F>   i `S   @`L?  l BCDEF @`?  l Be CDE,F6 \e @`Y?`  l 0By CDElFvR?.  %a3AA#Rcwr# $ 1>M[j{ (3?L[j z':K^p 7Vsu]C(l:l YY|o`RJA90& A80(!OA 5*("9JS\eo9BJRY^%b-e5f>fBeEeJdMbPaS_V]YV ^ d k p t w x y x u q zl kc [[ MQ ?F 3: (,   8<@`F?@  l Bz CDEXFdz y v q zl kd [\ MQ ?F 39 (-   R@.  %a3BA$Rcws$ $ 1?M[j{ )4@M\k {(:L^p 8Vsu]C(V _ e k p u x y z ]Y}oaRJB:0' B90("OA 5*)"9KS]fo9BKSY^%c-f5g>gBfEeJeMcPaS_V]Y/0@`@`^?B l B( CDEXFf}      # % ' ( ' $ z k [ M ? 3 (     y  0BS d!=Xqu]C(@qFhL^QTUJX?Z4[(\[ XSMD;0#=WpthYK ;$ +8 J \ n } T7*S j s | '09=@DHKNQT Y{riaWNE  %-5>BEJMPSV Y04@`@`@`{?0 l BCDEF w%n+c2Z9RBHJ@R9[2e*n$y ,7 DOZep${*29@HRZcnw*7BNZdoy{peZOD7,yne[RJB92+y%odZNB 7 *zrjaZSLE?y9o4f/\*R&H#< 2' #&*/49>zDpJhR`YWaPiJqCz<71-(% '2= G%R(e2x<JXhz'2<GQewxoe[RG=2'@`@` l BC%DEF EEQG>6-& {n%%f%%|vpiaYQXkw"%)/4:@GPYahntx|ziX@`@`H l 0޽h? ? 380___PPT10.@} Boeing2000  qi@p(  p p C "A B1bg"$ H@.WWWW p TEgֳgֳ ?" 2  W#Click to edit Master subtitle style$ $ p ZHgֳgֳ ?"p  T Click to edit Master title style! ! p  `Jg5%g5% ?"\z*  L Page *  p  `Ogg ?"    D*  p  `4Vgֳgֳ ?"``  B* H p 0޽h? ? 380___PPT10.@} 0 zr (    0 P    P*    0     R*  d  c $ ?    0h  @  RClick to edit Master text styles Second level Third level Fourth level Fifth level!     S  6 `P   P*    6$@ `   R*  H  0޽h ? ̙3380___PPT10.y 0$(  r  S _p0  r  S p`p`   H  0޽h ? ̙33  P 0(   x  c $|l0   x  c $$l2|K  H  0޽h ? ̙33  `0(  x  c $ܠl0   x  c $ܠl|K  H  0޽h ? ̙33  p0(  x  c $Pl0   x  c $(l 0  H  0޽h ? ̙33  0(  x  c $Lߠl0   x  c $l|K  H  0޽h ? ̙33   0(  x  c $-l0   x  c $x.l|K  H  0޽h ? ̙33   0(   x  c $l0   x  c $Ll|K  H  0޽h ? ̙33  `<(  `~ ` s *`Ӡl0   ~ ` s *Ӡl|K  H ` 0޽h ? ̙33  $0(  $x $ c $l0   x $ c $Xl|K  H $ 0޽h ? ̙33  @0(  @x @ c $l0   x @ c $l 0  H @ 0޽h ? ̙33  d<(  d~ d s *,l0   ~ d s *-l|K  H d 0޽h ? ̙33  P*(  Px P c $l0   r P S p1lP0  H P 0޽h ? ̙33  `0(  `x ` c $47l0   x ` c $7l|K  H ` 0޽h ? ̙33  <(  ~  s *)l0   ~  s *=l|K  H  0޽h ? ̙33  <(  ~  s *(l0   ~  s *H)l|K  H  0޽h ? ̙330   0(  x  c $+l0   x  c $\?l|K  H  0޽h ? f3Eh80___PPT10.P+0  00(  x  c $Jl0   x  c $Jl|K  H  0޽h ? f3Eh80___PPT10._10  @0(  x  c $(l0   x  c $l|K  H  0޽h ? f3Eh80___PPT10.bp@+  P<(  ~  s *Sl0   ~  s *Tl|K  H  0޽h ? ̙33  `<(  ~  s *[l0   ~  s *[l|K  H  0޽h ? ̙33  p<(  ~  s *Fl0   ~  s *$`l|K  H  0޽h ? ̙33  <(  ~  s *\jl0   ~  s *kl 0  H  0޽h ? ̙33  0(  x  c $nl0   x  c $\ # l 1?p5   I1  @` <\ # lH 1?p5  I5  @` :\ # l&  1?"`pP NOthers  @` 3\ # lT)  1?"` p  I9  @` 1\ # l7  1?"`5 p  R   @` /\ # l>  1?"`p5  R   @` -\ # l@  1?"`Pp ONo Data  @` \ # lH 1? `  K115  @` \ # l,Q 1?P   I1  @` \ # l^ 1? P  R   @` \ # l` 1?p  J19  @` \ # ln 1? p  I3  @`  \ # lp 1?<   J82  @`  \ # lx 1? <  IA  @`  \ # lH 1?5 `  J23  @`  \ # l 1?P 5   R   @`  \ # lh 1? 5 P  R   @` \ # lԝ 1?p5  I4  @` \ # l 1?5 p  R   @` \ # l 1?<5   J18  @` \ # l 1?5 <  IB  @` \ # l< 1?P 5  R   @`A \ # l  1?"`P P FP-Tool Limita-tion , @` \ # l 1?`5  J27  @` \ # l 1? P 5  R   @` \ # l 1?p 5  R   @` \ # l 1?p5  R   @` \ # lt 1?<5  J22  @` \ # l 1?<5  IC  @` \ # l8  1?"`P` MTotal  @`A \ # l\  1?"` PP  FN-Tool Limita-tion , @` \ # l  1?"`pP  VFalse Positive  @` \ # l   1?"`Pp VFalse Negative  @` \ # l  1?"`<P LPass  @` \ # l  1?"`P< R Test Level    @`B  \ To ?P`P~B !\ N1 ?`B "\ To ? ` B #\ To ?P ~B $\ N1 ?<P< ~B %\ N1 ?P ~B &\ N1 ?pPp ~B '\ N1 ? P ~B (\ N1 ?P PP B )\ To ?`P` ~B *\ N1 ?P ~B +\ N1 ?5 `5 ~B ,\ N1 ? ` ~B .\ N1 ?pPp ~B ;\ N1 ?P H \ 0޽h ? ̙33  ,<(  ,~ , s *8B l0   ~ , s * l 0   H , 0޽h ? ̙330  D0(  Dx D c $D l0   x D c $g l|K   H D 0޽h ? f3Eh80___PPT10./!xXol[W?I7n4-m֍dmN_]J'鈓 $ ?l~ZE^MBHI hc!ĄD?|O|@B v6`0{9\ :A.N Fxx4 r=x}c!0pQ@@P `8f8aggl{^O't@sWoۯ]d$$ՀO |njG-83dJb@E饒[/[SNU=o2KFRJU?W,^|bPDd-q<\v7M;n*{ޔ[(jݣwv}҄ߍ,ο&U`-GoS=UW"Z3 r Ι?F~]]gT(pUV $yW=HT_ #ԙ%vf⡄OH<RM|*Ѩ[S=jTIZ9w>-v#T_VGTF*V9rQy jl˔3gft_n>ﭵd8z.{}9u~x#vQ}c9jT/5vSG#g#koR0*A8ȔP{wT>W;r}mm맯ĵ\e0o  gюя>b'@'zH񣟾C~g3W~Zfӎ4?W;4"kȀl3\Z^c.![P8p1HS4 9O> $~'lNNMۜZa@4 K57''QSuhHY!HV¯bt#pnBC60ueZ0:s԰H? , VvY5߫@g~^'0ͬ7Z}sR_nqܕ+8b+1Gۿ6*D>ۅ i|CVN ­ۀ鰤,,^;:O mRd*2V\Z)E )GcF gs_:}FiZu]wOj)P6\nZܧ `Ր1FP$$Ê%@0K" %(4t|VM_%Reb!#\KaupQJw>d>(jY_p rd]y8޴rŏ}yk)P#É3 cg6@k۱D~LJLBHRFJV`u"FU[5Oh+'0  hp  $ 0 <HP@STUDY OF QUALIFICATION CRITERIA FOR SOFTWARE VERIFICATION TOOLS V. SanthanamIFI Boeing2000m air140kf0m591Microsoft PowerPointN C@@1@*|@`k{YGg   & &&#TNPP2OMi & TNPP &&TNPP    --- !---&Z"y.  --  *[    >-- ^{:8N @d K{  ?{ X * @cS$ X yUSj]lJ?R/Y uf i ir u 9 s G  0s 9 H r I N x  TD r v ~ g 8     | H     c6  @ q -  R | Y 6 Y  @R cu]?ua$%S<S2wHcR M @ N    Te   )nm(HL& ' * ? @(--3--$ m%4&#'$&'<"#$""m%--3--$*)y%*(+*)--3--$ 2;0a/-)+-V.0032--3--^$-:665%5k42w2k2(w2c223c77p88K6993Z:3 ;'7'66W65n5b5n557&c77p88882A8:r;;;~;H[;,;:--3--V8  Hi%kA(cLg\K   2C&,  2--3--a8 4Ho4l 2'Ky3yKV'4l<I`ll2a2--3--J8=9=<<`<<lt>>t>`:>===E=<`<<%]>ki>]>.>`==--3--48 >=H=19=19=UQ=UQ===U >U==>=Q=Q=H=H=T=k====----'&--yH--j}w@ U}ww0- @Times New Roman}ww0- 3.*2 STUDY OF QUALIFICATION  ! "! . 3.(2 CRITERIA FOR SOFTWARE  ! !, . 3."2 VERIFICATION TOOLS! !!.--q1@-- 3@Arialw@ \}ww0- .-2 ZMA report presented at the)  . .I2 s,2005 National Software & Complex Electronic    . .<2 #Hardware Standardization Conference     .@Arialw@ }ww0- . 2 27 . . 2 -7. . 2 July  . . 2 -7. . 2 2005 .@Arialw@ c}ww0- .2 Vdot Santhanam   . .*2 gPRINCIPAL INVESTIGATOR     . .2 !BOEING   .--"System 0-&TNPP &՜.+,D՜.+,<     On-screen ShowThe Boeing Companye Times New RomanArial Boeing2000@STUDY OF QUALIFICATION CRITERIA FOR SOFTWARE VERIFICATION TOOLSTOPICSOverview of ProjectPhase 1 FindingsPhase 2 FindingsStatement Coverage Issues"Decision Coverage Issues (1 of 2)"Decision Coverage Issues (2 of 2)MC/DC Issues (1 of 3)MC/DC Issues (2 of 3)MC/DC Issues (3 of 3)Resolving MC/DC Contradiction'Resolving MC/DC Contradiction (contd.)'Resolving MC/DC Contradiction (contd.)MC/DC Issues (3 of 3, contd.)Phase 3 ActivitiesTest Suite ObjectivesTest Suite Formulation Test Suite Formulation (contd.) Test Suite Formulation (contd.) Test Suite Formulation (contd.)Test Suite PrototypePhase 3 ResultsTool A ResultsTool B ResultsTool C ResultsCATS/p Preliminary Results Recommendations for Future Work  Fonts UsedDesign Template Slide Titles(8_AdHocReviewCycleID_EmailSubject _AuthorEmail_AuthorEmailDisplayNameÞڞ/Reminder: SW&CEH Material is due by COB 8 JulyCvdot.santhanam@boeing.comisSanthanam, Vdot _U air140kfair140kf  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~Root EntrydO)PicturesCurrent UserSummaryInformation(<PowerPoint Document(UDocumentSummaryInformation8