AssuranceCasesforSoftwareReleasesin ISSSustainingPhaseofDevelopment
Sarma SusarlaIV&V TeamLeadSarma.V.Susarla@nasa.gov9/10/2013
Introduction
IV&V has been performing ISS IV&V for the past 18 yearsSoftware architecture and design is matureHowever new releases are being developed to add new functionality and product improvements in the existing architecture.IV&V had been generating technical reports at the end of each milestone event such as :Technical Design Review (TDR)Test Readiness Review (TRR)Software Transition Readiness Review (STRR)These reports document software assurance stating that the CSCI is ready for next stage in life cycle development.This presentation provides a framework for an integrated software assurance approach:To assert that CSCI is ready for on-orbit deploymentUsing existing IV&V processes and analyses.
Sarma Susarla IV&V Workshop 9/10/2013
2
Evidence Based Assurance (EBA) ImplementationApproach
Use ISS life cycle development modelUselife cycle developmentmilestonesasindividual claimsinto EBA networkmodelRequirements development completionCode development completion etc.UseBiwarreport as a mechanism to report evidenceas it becomes available during life cycle development support activityMaximize usage of existing evidence data from technical reportsMinimize new documentation workExisting analyses remain same but may become deeperand more rigorous to support each claim
Sarma Susarla IV&V Workshop 9/10/2013
3
ISS CSCI Life Cycle
In normal waterfall mode, the life cycle Sequence is SRR, PDR, CDR, TRR, and FQT.In ISS above model is used for CSCI first release.Subsequent CSCI releases every 15 months toEnhance functionality of existing functionsAdd new mission requirements such as Visiting Vehicle supportFix existing critical software problemsImplement code changes to avoid operator workarounds (SPNs) for code problemsIn sustaining phase ,each release is an incremental change to the previous releaseOnly changes to the review artifacts are subject to formal reviewsTo cut down development time, parallel development activity for:Requirements developmentDesign/Code developmentFQT test development
Sarma Susarla IV&V Workshop 9/10/2013
4
LifecycleTimeline
RCS Review Process
Test design development
Design Test Review Process
Test Implementation- Test Script Development
Implementation Test Review Process
Software design/code development
Peer Review Process
Requirements Development
6 months
6 months
Content Review
TDR
TRR
2 months
Sarma Susarla IV&V Workshop 9/10/2013
StageTRRs
FQT end
Stage End
STRR
CSCIreadyfor on-orbit transition
LegendTDR: Technical Design ReviewTRR: Test Readiness ReviewFQT : Formal Qualification TestingSTRR: Software Transition Readiness ReviewRCS: Requirement Change Sheet
4 months
5
CSCI Life CycleReviews
Sustaining engineering CSCI development life cycleContentreviewSCR(Software Change Request)database contains desired new functionality, existing problems, and product improvement suggestionsSCR Content is selected based on community high priority selectionsIV&V selects open SCRs from the databaseTo meet mission requirementsTo fix critical problemsRequirement Change ReviewsOn going, held after requirement changes for each SCR are workedIV&V evaluates changes for Q1, Q2, Q3 and provides issue feedbackTechnical Design Review (TDR)Held after all requirement changes and new designs are completeCombines SRR, PDR, andCDR in waterfall modelIV&V evaluates design changes for Q1, Q2, Q3 and provides issue feedbackCode ReviewsOn going, held after code developed for each SCR triggered changeIV&V evaluates changes for Q1, Q2, Q3 and provides issue feedbackTest Design ReviewsOn going, held after test case designs to verify new requirements are developedIV&V evaluates design to ensure that the tests verify the requirements correctly and completely.Q2 and Q3 considered as applicable
Sarma Susarla IV&V Workshop 9/10/2013
6
CSCI Life Cycle Reviews
TestImplementation ReviewsOn going, review held after a test script for an FQT test is developed anddryrunon FQT platformIV&V verifies from test log and script that the test is implemented as per test design and test cases ran in the expected sequence and orderIV&V also verifies that test failures are captured in SCRs.Test Readiness Review (TRR)Held after all tests completed test implementation reviews, review issues were incorporated, and tests weredryrunon flight release.IV&V verifies that there are no open issues on any test cleared for FQT runFormal Qualification Test ResultsReviewIV&V reviews test logs to verify that tests ran as expected and test failures are properly described in referenced SCRs.Stage Test ReviewsIV&V reviews integrated tests that verify the CSCI performance in the on-orbit CSCIs integrated environmentIV&V ensures that test procedures match test objectives and test failures are correctly reported in SCRsSoftware Transition Readiness ReviewIV&V evaluates open SCRs/Issues to ensure that the corresponding problem does not impact on-orbit CSCI transition to new release or immediate operation with new releaseIf such SCRs/Issues are found, IV&V would articulate to the Program to address the problem before the transition.
Sarma Susarla IV&V Workshop 9/10/2013
7
Proposed CSCI Assurance Model
ComponentsA Generic assurance claim networkThe Generic model is instantiated for each CSCI to suit the CSCI’s specific life cycle specifics.Sub-claims may be added to suiteListed evidence in the model may be substituted with different evidence applicable evidenceTBDs in the model will be replaced with actual numbersA Generic evidence reporting mechanismAnalysis DriversTechnical Reference for Adverse ConditionsTechnical Reference for undocumented CSCI behaviors as neededPAL assets as extensions to Catalog of Methods for ISS specific artifacts analysis for:RequirementsSoftware designFlight codeTest design and implementation
Sarma Susarla IV&V Workshop 9/10/2013
8
Assurance Model Concept
Inverted Tree structure of various claims each claim representing completion of a life cycle development eventTop most claim represents final assurance claim for Software on-orbit deploymentFor each Sub-ClaimClaim StatementSecond level claims representing each major milestone in software development at Process level E.g.Requirements development completeCode development complete etc.Third level claim represents sub-claims if second level claim is made up of several distinguishable life cycle eventsThe last level claim represents analysis activity requiring certain rigor such as IV&V 3 QuestionsEvidenceMost of the data is what is generated for existing technical reportsTDR ReportTRR reportSTRR reportList of developer products such as SCRs, RCSs and Tests reviewed and IV&V issuesLifecycle analysis activity and resultsreportedinBiwarFor activity which is outside the scope of IV&V (such as simulation, unit tests etc), use developer’s status reports as supporting evidenceArgumentDescription of life cycle analysis activity pertinent to realizing the claimContains reference to PAL assets and technical reference as appropriate
Sarma Susarla IV&V Workshop 9/10/2013
9
Assurance Case Claim NetworkMost Claims supported by IV&V Life cycle review activity
Code behavior verified by unit and integration tests
All CSCI issues that impact on-orbit transition have been addressed
FQT tests Verified CSCI behavior
CSCI behavior verified in integrated on-orbit configuration
1.5
1.7
1.8
All tests completed dry runs using flight software on FQT platform
TRR established CSCI, SIM and test rig readiness for formal testing
CSCI behavior verified through formal testing
1.6.4
1.6.5
1.6.6
Test designs are adequate to verify requirement changes
Test scripts and procedures implemented the designs accurately
Regression tests were adequate
Test designs verified as per IV&V 3 questions
1.6.1.1
1.0
SCR content represents mission concept
Design adequately represents requirements functionality
Code developed to match requirements and SCRs
Requirements developed per SCR content
1.1
1.2
1.4
Requirements evaluated per IV&V 3 Questions
Design evaluated per IV&V 3 Questions
Code evaluated per IV&V 3 Questions
1.4.1
1.2.1
1.3.1
Final claim at STRR
1.3
1.6
1.6.1
1..6.3
1.6.2
Each Box isa claim
CSCI ready for on-orbit transition
Sarma Susarla IV&V Workshop 9/10/2013
10
Entire claim network is a word document containing for each claim: Claim statement; Evidence, Argument , Caveats , Supplemental info
Analysis Rigor
MethodsAnalysis methods expanded from COM methods to detailed ISS specific guidelines.Separate PAL asset each for Requirements, Design, Code and Test.Adverse conditionsFull list of adverse conditions captured in TR folder in ECM, next slide shows a shortened listThese conditions are evaluated as applicable to Requirements, Design, Code, and Test reviews with respect to Q2, Q3.If an adverse condition is integrated into a requirement, then further life cycle analysis for that will be under Q1.There will be several adverse conditions in code that will not be captured in requirements. Code will be reviewed against such adverse conditionsVery few adverse conditions for test analysis because FQT will not test behaviors not captured in requirementsAdverse conditions not captured in requirements will be topics for Independent Analysis(IA).
Sarma Susarla IV&V Workshop 9/10/2013
11
ISS Adverse Conditions (Partial list)
Sarma Susarla IV&V Workshop 9/10/2013
12
Evidence Reporting
Next set of slides contain reporting samples for some assurance claims in the claim networkReported as part ofBiwaras claims are realized during the life cycle developmentAt the end of the life cycle, a technical report is prepared containing:claim networkIV&V activity and evidence that supports each claim
Sarma Susarla IV&V Workshop 9/10/2013
13
Requirements Claim (1.2)
ClaimRequirements are developed as per CSCI SCR content and completed development prior to design and code developmentEvidenceList of RCSs developed, and reviewed by IV&V and othersBiwarreport confirming that Review issues were incorporated into the requirements and requirements development is completedIV&V RCS review status reported inBiwarRCS status reported at TDR(Sub-Claim) Requirement evaluation per IV&V 3 QuestionsArgumentEach RCS is reviewed against the corresponding SCR to ensure that the requirement adequately captures the SCR intentReview issues were submitted anddispositionedwith reviewer’s concurrenceRequirements development is completed by TDR except for post-TDR SCR content changes where the requirements development is completed before their design/code implementation.
Sarma Susarla IV&V Workshop 9/10/2013
14
Requirements AnalysisSub-claim (1.2.1)
ClaimThe requirements adequately capture what the system is supposed to do, what it is not supposed to do and how the system should behave under adverse conditions for defined conditionsEvidenceList of RCSs and the SCRs driving changes to requirementsList of accepted issues submitted by IV&VUsage of PAL asset “ ISS RequirementsReview and AnalysisGuidelines” for analyzing the requirementsArgumentIV&V analyst is instructed to use the guidelines in the PAL asset for requirements analysis.Each requirement is analyzed to match the concept in the SCR that drives the new requirement or change to the existing requirement.Issues were submitted to the developer where the requirement is deficient from IV&V three questions perspective.The analysis will cover all aspects for IV&V Q1 and practical situations for IV&V Q2 and Q3.Supplemental informationAdverse conditions to be evaluated for are listed in ISS Technical Reference folder. The PAL asset contains specific guidelines designed for ISS development context. These guidelines provide instructions on how to verify for IV&V Q1, Q2, and Q3.CaveatsFor IV&V Q2, it is not possible to specify what the software is not supposed to do (negative specifications) for all cases and hence this specification is limited. During requirements analysis, IV&V will ensure that negative specifications are provided where practical.For IV&V Q3, it is not possible to specify software actions for all possible adverse conditions However, certain adverse conditions (such as communication error, stale data, data exceeding a threshold, software timeout etc. can be integrated into requirement specifications .
Sarma Susarla IV&V Workshop 9/10/2013
15
Requirements ClaimReporting
Reported after SRS incorporated all requirement changes as specified in content SCRs, usually after TDR (combines 1.2 and 1.2.1)CSCI requirements development is completed and the new requirements are captured in SRS version<tbd>,in time for design and coding to proceed. Requirements were developed through<tbd>RCS reviews.IV&V evaluated the requirements to ensure that they capture what the software is supposed to do as indicated in the respective SCRs. IV&V also evaluated the requirements to ensure that as specified, the software does not do what it is not supposed to do and behaves adequately under applicable adverse conditions as indicated in ISS Technical reference “adverse conditions”.All accepted review issues were incorporated correctly into the SRS.Evidence attachments to Technical reportTable showing SCR #, RCS number, review dateList of IV&V’s accepted issuesTable showing for each SRS function, # of new/changed requirements
Sarma Susarla IV&V Workshop 9/10/2013
16
Code DevelopmentClaim (1.4)
ClaimCode development required by the SCR content is completedEvidenceStatus report inBiwarof IV&V Code reviews, either in developer’s peer reviews or separately and incorporation of review issues.List of accepted IV&Vissues/ or SCRsgenerated during code reviews(Sub-claim) Code evaluation per IV&V 3 questionsArgumentIV&V reviewed code in code peer reviews and provided issue feedback.If peer reviews were not held or missed, IV&V reviewed code changes after the new release code is availablePeer review issues were submitted to the developer during peer reviews andthe acceptedissues were incorporated into the final code.If code review is done after the code is formally released, any issues found during the review are reported as SCRs and the SCR process ensures that the issue is appropriately addressed via seal-break process if the problem warrants such action;otherwisethe issue will be fixed in a future release.Supplemental informationTools used to perform code analysis included Understand Ada, Understand C, andTextpad.
Sarma Susarla IV&V Workshop 9/10/2013
17
CodeEvaluation Sub-claim(1.4.1)
ClaimCode implemented correctly to produce behavior desired by the requirements and/or SCR on what it is supposed to do, what it is not supposed to do and perform adequately under applicable adverse conditions.EvidenceBiwarreport indicating code review completion by IV&V for all code changing SCRs in the releaseUsage of PAL asset “ISS CodeAnalysisGuidelines” for code reviewArgumentIV&V analyst is instructed to use the guidelines in the PAL asset for code analysis.The code is evaluated to ensure that it implements the design documented in the design document and/or documented in requirementsFor those code change SCRs that do not have a corresponding requirement change, the code is evaluated toensure that it implementsthe behavior or code correction as specified in the SCR analysis recommendation.Issues were submitted to the developer where the design does not match functionality desired from the requirement.The code analysis will cover all aspects for IV&V Q1, Q2, Q3 where formal requirements exist. For those cases where formal requirements do not exist, IV&V will evaluate the code for all practical and credible aspects of Q2 and Q3 code scenarios that can take place during code execution; these practical scenarios are specified in the code analysis guidelines.Supplemental informationUnderstandADA/C,Textpadtools are used to navigate through the code and perform code analysis
Sarma Susarla IV&V Workshop 9/10/2013
18
Code DevelopmentClaimReporting
Reported after new flight code is released and IV&V has completed code review (combines 1.4 and 1.4.1)IV&V reviewed code changes made for this release and confirmed thatthe code implements what is required as per the SCRs,respective requirements and the design documents TLDD, DBDD, and SUM. IV&V alsoevaluated the code changes to ensure that the code performs adequately under applicable adverse conditions as listed in ISS technical reference“ISS adverse conditions” and doesnot cause any unwanted behavior. All accepted code issues were implemented in the code.Evidence attachments to technical reportList of code change SCRs, corresponding review dateList of accepted IV&V issues
Sarma Susarla IV&V Workshop 9/10/2013
19
FQT CompletionClaim (1.6.6)
ClaimRequired CSCIbehavior is verified though formal tests and test anomalies were reported correctly in SCRsEvidenceList of FQT tests, pass/fail status, corresponding SCR for failed testsBiwarreport indicating IV&V evaluation of FQT resultsArgumentIV&V evaluated test logs from formal testing to ensure all defined FQT tests were executedIV&V examined the test failures and evaluated the corresponding SCR that reported the failure to make sure that the SCR correctly describes the failure.Supplemental informationProgram board, SCSRP, evaluates the impact of each SCR on ISS on-orbit operation and authorizes software changes via PPLs or patches to supplement the flight software for on-orbit usage.IV&V provides feedback to the Program in the SCRdispositioningprocess
Sarma Susarla IV&V Workshop 9/10/2013
20
FQT completion reporting
Reported after IV&V completed evaluation of all FQT test results (for 1.6.6)CSCI Formal Qualification Testing completed. From the test reportIV&V verified, that all planned FQT tests were run.<tbd1>of a total of<tbd2>,tests passed confirming that the software is performing as per the missionrequirementsexcept for the requirements listed below.For the failed tests IV&V verified that the test failure is correctly reported in the referenced SCR. TheCSCI life cycle development is completed and the CSCI is verified to perform as required for the mission, with the exception of the followingrequirements.These failures will be assessed by the Program through the SCSRP process and appropriate changes will be made to the CSCI via PPL/Patch as needed for on-orbitoperationEvidence attachments to technical reportList of FQT tests with pass/fail status and associated SCR info for failed tests
Sarma Susarla IV&V Workshop 9/10/2013
21
CSCI Transition Readiness Claim (1.8)
ClaimAll software issues that affect on-orbit transition of the new release have been fixedEvidenceList ofundispositionedSCRs on the CSCI along with IV&V assessment as reported inBiwarList ofdispositionedSCRs that are targeted for fix in a future release along with IV&V assessment as reported inBiwarIV&V’s assessment of work status of all review issues and SCRs targeted for fix in the current release as reported inBiwarIV&V’s assessment of completion of SPNs targeted for this release as reported inBiwarArgumentIV&V evaluates allundispositonedSCRs on this release to determine their impact on software behavior for software transition or immediately following transitionIV&V evaluates all SCRs that are targeted for fix in a future release to determine their impact on software behavior for software transition or immediately following transitionIf any of those SCRs affect safety critical on-orbit operation, IV&V will request the SCSRP board to evaluate those SCRs prior to on-orbit software transition to determine if any patches or PPLs need to be developed.IV&V evaluates any open issues from the CSCIreviews or SCRs targeted for fix in the current releaseto determine their impact onon-orbittransition and immediate software.IV&V reviews SPNs developed for the release to make sure that operational workaround is documented
Sarma Susarla IV&V Workshop 9/10/2013
22
Transition readiness reporting
Report this after the CSCI readiness is evaluated for on-orbit transition (1.8)IV&V completed on-orbit transition readiness review and no exceptions have been uncovered. IV&V evaluated<tbd1>undispositonedSCRs on the current and previous releases and<tbd2>dispositionedbut targeted for fix in a future CSCIrelease.IV&Vanalyzed these SCRs and determined that they do not impact the software operation for successful on-orbit transition from the previous release and immediate on-orbit operation with the new release. All work targeted for fix on this release from SCRs and milestone review issues is completed.IV&V verified that all SPNs, numbering<tbd>,triggered by the new release have been developed. IV&V’s analysis concludes that the CSCI is ready for on-orbit transition.Evidence attachments to technical reportList ofundispositionedSCRs showing SCR #, Title, CSCI Status, IV&V evaluation commentList ofSCRs approved for future work showing SCR#, Title, CSCI. Status, IV&V evaluation comment
Sarma Susarla IV&V Workshop 9/10/2013
23
CSCI Technical report
Final Technical Report is prepared and will be delivered at Transition Readiness ReviewReport will focus on the EBA network model and realization of claimsFor data, report relies heavily on using what is reported inbiwarand internal worksheets currently generated during IV&V life cycle work.Report FormatOverall claim tree chart Plus for each major claim in the overall claim chart, the following:Claim statement, Evidence, Argument, Caveats, supplemental information (as given in previous slides)IV&V statement on claim realization as reported inBiwar(as given in previous slides)Evidence data attachments
Sarma Susarla IV&V Workshop 9/10/2013
24
Assurance cases for software releases in ISS sustaining phase of development
Questions/Comments
Sarma Susarla IV&V Workshop 9/10/2013
25
0
Embed
Upload