[{"name":"R5-246542","title":"AI ML Conformance Call flow","source":"Qualcomm Incorporated","contact":"Yogesh Tugnawat","contact-id":63855,"tdoctype":"discussion","for":"Discussion","abstract":"As per RP-234039 from RP#102@Dec 2023, in the WID of R19 WI \u201cNR_AIML_Air\u201d, for the \u201cTestability and interoperability\u201d topic, RAN5 aspects related to testability and interoperability to be addressed on a request basis. _x000D_\nRAN5 has initiated a dedicated agenda on AI ML testability aspects with an effort to help core groups with areas of RAN5 expertise. _x000D_\nChallenge(s) exists on conformance testing of LCM of AI ML from UE\u2019s initial activation and starting configuration to effectively test core requirements to prove gains in performance in a reliable and repeatable way._x000D_\nThis Proposal aims to introduce a call flow which might be adaptable to various needs of AI ML Testability and might be able to contain some aspects of design in RAN5._x000D_\nIn this proposal we have examined the UE Sided Beam Management Testability i.e one sided model, but other features can be similarly adapted which may need 2 sided models ex- CSI Compression._x000D_\nDISCLAIMER\u00a0\u2013 Due to the dynamic nature of discussions within various core groups, this discussion\\proposal should not be regarded as RAN5 feedback for any other core group or referenced in any feature discussions within core groups._x000D_\n_x000D_\nProposal \u2013 AI\\ML Test Mode :_x000D_\n3GPP RAN5 uses Test Mode (Test Loopback Mode or Test Functions) to put UE in a specific test mode for testing. These are specified inTS38.509 to be implemented in NAS layer for NR._x000D_\nOnce Core Requirements are finalized, proposal is to introduce a new Test Loop Back Mode (TLB Mode), Test Commands\\Functions(s), as needed, in TS 38.509, which may \u2013_x000D_\nActivate and Deactivate UE in AI ML Test Mode to ensure UE\u2019s is prepared with out any interference_x000D_\nDefine the selection of AI ML feature for UE along with initialization and trigger of assistance data to the UE\u2019s AI ML Model for given functionality ; this can be fully defined in RAN5 to be uniform among UE and TE implementation. _x000D_\nNW side AI ML Feature can be included in same Test Mode, as and when the requirements are defined._x000D_\nAny other function which is required to prepare the UE for AI ML Test Evaluation._x000D_\nThe purpose of Test Mode will not be \u2013_x000D_\nTo assist UE in any way shape or form which may show skewed results in UE favor_x000D_\nTo aid UE in controlled environment which may not be representative of real-world scenario_x000D_\nAn example of appropriate usage of test modes in Possible AI ML Call Flow for Conformance with UE side Beam Management Case-1 is presented in Qualcomm contribution in R5-247151","secretary_remarks":"","agenda_item_sort_order":1220,"ainumber":"4.2.3","ainame":"AI\/ML Testability & Interoperability","tdoc_agenda_sort_order":65420,"status":"noted","reservation_date":"2024-06-11 23:07:49","uploaded":"2024-11-22 22:11:04","revisionof":"","revisedto":"","release":"","crspec":"","crspecversion":"","workitem":"","crnumber":"","crrevision":"","crcategory":"","tsg_crp":"","lsreplyto":"","lsto":"","Cc":"","lsoriginalls":"","lsreply":"","link":"https:\/\/www.3gpp.org\/ftp\/TSG_RAN\/WG5_Test_ex-T1\/TSGR5__105_Orlando\/Docs\/R5-246542.zip","group":"R5","meeting":"R5-105","year":2024,"uicc_affected":"","me_affected":"","ran_affected":"","cn_affected":"","clauses_affected":"","crsinpack":null,"crsinpacknumber":0},
{"name":"R5-246707","title":"Discussion on handling of AI\/ML in RAN5","source":"CMCC, AT&T, CAICT, Ericsson, Huawei, MediaTek, Nokia, ZTE","contact":"Dan Song","contact-id":86282,"tdoctype":"discussion","for":"Endorsement","abstract":"Related R4 Tdoc is R4-2418457._x000D_\nObservation 1: During RAN4#112bis,  \u201chow to conduct the validation of a change in functionality in already deployed UEs\u201d was discussed under \u201cIssue 1-2: Post deployment testing\u201d with the following agreements captured in R4-2417212 [1]._x000D_\nObservation 2.0: GCF-AP[2] Section 7.7 covers the scenario of \u201chow to deal with the changes in Product\u2019s Hardware and\/or Software Configuration\u201d, which addresses the scenario of \u201chow to conduct the validation of a change in functionality in already deployed UEs\u201d from a methodology perspective. _x000D_\nIt is feasible to handle this for the changes in already certified\/deployed UEs._x000D_\nObservation 2.1: For the UE which already supports AI functionality\/features when it is certified, and the certified AI-UE is subsequently modified with updated AI models, as long as the changes in the AI models do not add new functionality\/features, this scenario is aligned with: \u201cdo not add new functionality or features\u201d described in GCF-AP[2] Section 7.7. _x000D_\nEvidence of continued compliance with the applicable Certification Criteria shall be provided and recorded by the certification bodies._x000D_\nObservation 2.2: The scenarios below are addressed by \u201cadd new functionality or features\u201d described in GCF-AP[2] Section 7.7._x000D_\nThe UE does not support AI functionality\/features when it is certified, and the certified UE is subsequently modified with addition of AI functionality\/features_x000D_\nThe UE already supports AI functionality\/features when it is certified, and the certified UE is subsequently modified with updated AI models which enable the certified UE with new functionality\/features_x000D_\nEvidence for compliance to the Certification Criteria applicable to the new functionality for feature shall be provided and recorded by the certification bodies._x000D_\nObservation 2.3: For the change that requires a modification to the information within the declaration, the certification bodies shall be notified in time, including both the scenarios below described in GCF-AP[2] Section 7.7_x000D_\n \u201cdo not add new functionality or features\u201d: The UE already supports AI functionality\/features when it is certified, and the certified AI-UE is subsequently modified with updated AI models. However, the changes in the AI models do not add new functionality\/features _x000D_\n \u201cadd new functionality or features\u201d: _x000D_\nThe UE does not support AI functionality\/features when it is certified, and the certified UE is subsequently modified with addition of AI functionality\/features_x000D_\nThe UE already supports AI functionality\/features when it is certified, and the certified UE is subsequently modified with updated AI models which enable the certified UE with new functionality\/features_x000D_\nObservation 3.1: There are a few cases tagged with \u201cUpdated\u201d can be found in GCF website. For an updated device, some manufacturers may choose to make a new certification rather than updating the original. _x000D_\n Observation 3.2: In GCF, there is no additional fee for doing a new certification or updating a certification. But Recognised Test Organisation (RTO) may still charge for doing a new certification or updating a certification. _x000D_\n_x000D_\nProposal 1: Considering RAN5 is familiar with the existing certification\/validation methodology\/processes in the certification bodies (quite a few RAN5 delegates attend both RAN5 and GCF), RAN5 should discuss methods to address how to conduct the validation of changes in functionality in already deployed UEs relevant to \u201cIssue 1-2: Post deployment testing\u201d as part of \u201cAI\/ML Testability and Interoperability\u201d. _x000D_\nProposal 2: RAN5 take into consideration the methodology of \u201chow to deal with the changes in already deployed UEs in GCF\u201d as a starting point with necessary adaptions when discussing \u201chow to conduct the validation of a change in functionality in already deployed UEs\u201d under \u201cIssue 1-2: Post deployment testing\u201d.","secretary_remarks":"","agenda_item_sort_order":1220,"ainumber":"4.2.3","ainame":"AI\/ML Testability & Interoperability","tdoc_agenda_sort_order":67070,"status":"revised","reservation_date":"2024-07-11 10:34:36","uploaded":"2024-11-08 15:34:00","revisionof":"","revisedto":"R5-247647","release":"","crspec":"","crspecversion":"","workitem":"","crnumber":"","crrevision":"","crcategory":"","tsg_crp":"","lsreplyto":"","lsto":"","Cc":"","lsoriginalls":"","lsreply":"","link":"https:\/\/www.3gpp.org\/ftp\/TSG_RAN\/WG5_Test_ex-T1\/TSGR5__105_Orlando\/Docs\/R5-246707.zip","group":"R5","meeting":"R5-105","year":2024,"uicc_affected":"","me_affected":"","ran_affected":"","cn_affected":"","clauses_affected":"","crsinpack":null,"crsinpacknumber":0},
{"name":"R5-246818","title":"On AI\/ML Testability Discussions in RAN5","source":"Keysight Technologies UK Ltd, Anritsu Corporation, Rohde & Schwarz","contact":"Flores Fernandez Martos","contact-id":47301,"tdoctype":"discussion","for":"Endorsement","abstract":"This document provides company views on how RAN5 should proceed regarding AI\/ML discussions. TE vendors and many other companies have been actively participating in associated discussions in RAN4. By no means is the intention of this paper to slow down the progress on AI\/ML testability discusssion. However, Keysight\u2019s observations and proposal present different views than those of the RAN5 leadership. _x000D_\nObservation 1: NR AI\/ML for air interface WID [2] clearly states that the responsibility for testability and interoperability aspects lies with RAN4 while RAN5 aspects related to testability and interoperability were to be addressed on a request basis._x000D_\nObservation 2: There has been no explicit request from RAN4 to RAN5 to address any testability\/interoperability aspects._x000D_\nObservation 3: Keeping testability discussions in RAN4 as part of the WI allows requirements definition to consider testability aspects and vice versa which was considered essential by RAN Plenary. _x000D_\nObservation 4: AI\/ML for NR air interface testability discussions have been held for almost 3 years already in RAN4 (since Rel-18 SI for AI\/ML was triggered) and will continue as part of Rel-19 corresponding WI while requirements are being defined._x000D_\nObservation 5: Replicating AI\/ML testability discussions in RAN5 is not efficient and could lead to different conclusions in different working groups._x000D_\nObservation 6: Guidance from RAN5 leadership might be beneficial which testability aspects should be discussed in RAN5 at this stage that are not discussed in RAN4 given the lack of progress in the requirements definition._x000D_\nObservation 7: Companies can provide their views through their RAN4 delegates or even RAN5 delegates attending RAN4 AI\/ML sessions._x000D_\nObservation 8: RAN5 is not in a position now to make technical contributions to RAN4 specs covering any of the objectives in the WID and hence, RAN1\/2\/3\/4 AI\/ML WI for NR air interface should not be updated to add RAN5 secondary responsibility._x000D_\nProposal 1: It is proposed for RAN5 not to intiate detailed testability AI\/ML discussions under specific RAN5 agenda item until triggered by RAN\/RAN4, e.g., LS from RAN4, creation of RAN5 WI, RAN5 being assigned secondary responsibility of existing AI\/ML WI.","secretary_remarks":"","agenda_item_sort_order":1220,"ainumber":"4.2.3","ainame":"AI\/ML Testability & Interoperability","tdoc_agenda_sort_order":68180,"status":"noted","reservation_date":"2024-07-11 16:56:12","uploaded":"2024-11-07 17:01:55","revisionof":"","revisedto":"","release":"","crspec":"","crspecversion":"","workitem":"","crnumber":"","crrevision":"","crcategory":"","tsg_crp":"","lsreplyto":"","lsto":"","Cc":"","lsoriginalls":"","lsreply":"","link":"https:\/\/www.3gpp.org\/ftp\/TSG_RAN\/WG5_Test_ex-T1\/TSGR5__105_Orlando\/Docs\/R5-246818.zip","group":"R5","meeting":"R5-105","year":2024,"uicc_affected":"","me_affected":"","ran_affected":"","cn_affected":"","clauses_affected":"","crsinpack":null,"crsinpacknumber":0},
{"name":"R5-247083","title":"On Testability and Interoperability Aspects for AI\/ML Based Beam Management","source":"Nokia","contact":"Siddharth Das","contact-id":99463,"tdoctype":"discussion","for":"Discussion","abstract":"In this paper, we share our views on potential open RAN4 issues related to the testability and interoperability of AI\/ML-enabled BM-Case1 and BM-Case2. Specifically, we cover the following aspects:_x000D_\n-\tTesting setup and Channel models_x000D_\n-\tNew UE Conformance Test Function for AI\/ML use cases_x000D_\nThe following Observations and Proposals were made:_x000D_\nObservation 1: CDL channel models present a more realistic way for testing AI\/ML BM use cases due to the more realistic spatial behaviour._x000D_\nObservation 2: As the FR2 channel is spatially sparse, a simplified CDL model with reduced number of clusters will be a good approach for the conformance testing of the AI\/ML-enabled BM use case. _x000D_\nProposal 1: RAN5 to suggest RAN4 to decide the number of clusters and their angular location in the simplified CDL model considering the following aspects:_x000D_\n(i)\tA practical number of test probes to be made available in the OTA test chamber for AI\/ML-enabled BM UE conformance testing,_x000D_\n(ii)\tThe angular location of the selected clusters should span the maximum part of the azimuth and zenith domains._x000D_\nObservation 3: It is not very clear whether all Set A\/B beams can be emulated inside the test chamber or only some of those beams that are \u201cmost relevant\u201d Tx beams can be emulated._x000D_\n  Proposal 2: RAN5 to discuss on below open issues regarding test setup:_x000D_\n1.\tIs UE rotation during the test required or not as it shall impact on maximum number of set A TX beams that can be generated inside the chamber?_x000D_\n2.\tBased on cost, chamber size, etc., what can be the \u201cmost relevant\u201d Set A Tx beams that RAN WG5 can agree upon considering closer to field scenario?_x000D_\nObservation 4: The existing test function isn\u2019t sufficient to handle the Rel-19 AI\/ML use cases and modes._x000D_\nProposal 3: RAN5 to further discuss a new UE special conformance test function to cater to AIML-enabled functionalities.","secretary_remarks":"","agenda_item_sort_order":1220,"ainumber":"4.2.3","ainame":"AI\/ML Testability & Interoperability","tdoc_agenda_sort_order":70830,"status":"noted","reservation_date":"2024-08-11 10:51:48","uploaded":"2024-11-08 15:44:04","revisionof":"","revisedto":"","release":"","crspec":"","crspecversion":"","workitem":"","crnumber":"","crrevision":"","crcategory":"","tsg_crp":"","lsreplyto":"","lsto":"","Cc":"","lsoriginalls":"","lsreply":"","link":"https:\/\/www.3gpp.org\/ftp\/TSG_RAN\/WG5_Test_ex-T1\/TSGR5__105_Orlando\/Docs\/R5-247083.zip","group":"R5","meeting":"R5-105","year":2024,"uicc_affected":"","me_affected":"","ran_affected":"","cn_affected":"","clauses_affected":"","crsinpack":null,"crsinpacknumber":0},
{"name":"R5-247151","title":"AIML performance use cases","source":"Qualcomm Inc","contact":"Vijay Balasubramanian","contact-id":84569,"tdoctype":"discussion","for":"Information","abstract":"DISCLAIMER \u2013 Several aspects of overall Core & Performance requirement are FFS in core groups_x000D_\n_x000D_\n- Beam Management_x000D_\nMain focus of the WI in RAN4_x000D_\nContent intended for RRM spec 38.133. FR2 only at this time._x000D_\n- Positioning accuracy enhancements_x000D_\nVery few aspects discussed by RAN4 at this time. _x000D_\nWhich spec? TBD_x000D_\n- CSI feedback enhancement (CSI prediction)_x000D_\nUE sided model_x000D_\nA version of PMI reporting test case being discussed_x000D_\nThroughput ratio as the KPI_x000D_\nFR1 is conducted. FR2 is radiated with wireless cable approach _x000D_\nContent intended for DEMOD spec 38.101-4. _x000D_\n- CSI compression (2 sided model)_x000D_\nPlanned to be in TR phase_x000D_\n_x000D_\nObservation 1: RAN4 is far from agreeing on the requirements for any of the AI ML use case._x000D_\nAs part of core requirements discussion, basic framework will likely be agreed first_x000D_\nFiner test case level details will happen much later during performance phase. _x000D_\nSimulation link level assumptions currently agreed to investigate impact of measurements error on prediction accuracy_x000D_\n_x000D_\nProposal1: For each of the AIML performance use case verification, UE test function is needed to activate\/deactivate AIML functionality in the DUT. _x000D_\nProposal2:  RAN5 can use its MU\/TT expertise to provide TT analysis for the initial set of parameters being considered by RAN4 to see the feasibility of those params. Especially the fading aspects which will be new for BM cases.","secretary_remarks":"","agenda_item_sort_order":1220,"ainumber":"4.2.3","ainame":"AI\/ML Testability & Interoperability","tdoc_agenda_sort_order":71510,"status":"noted","reservation_date":"2024-08-11 12:17:11","uploaded":"2024-11-22 22:11:04","revisionof":"","revisedto":"","release":"","crspec":"","crspecversion":"","workitem":"","crnumber":"","crrevision":"","crcategory":"","tsg_crp":"","lsreplyto":"","lsto":"","Cc":"","lsoriginalls":"","lsreply":"","link":"https:\/\/www.3gpp.org\/ftp\/TSG_RAN\/WG5_Test_ex-T1\/TSGR5__105_Orlando\/Docs\/R5-247151.zip","group":"R5","meeting":"R5-105","year":2024,"uicc_affected":"","me_affected":"","ran_affected":"","cn_affected":"","clauses_affected":"","crsinpack":null,"crsinpacknumber":0},
{"name":"R5-247502","title":"Discussion on AI ML testing","source":"Apple Benelux B.V.","contact":"Ashwin Mohan","contact-id":102945,"tdoctype":"discussion","for":"Endorsement","abstract":"Observation 1: The feasibility of RAN5 contributions to the test setup and methodology section of beam management work in RAN4 is pending assessment_x000D_\n_x000D_\nProposal 1: A clear framework for potential RAN5 contributions on the AI\/ML BM OTA discussion is critical \u2013_x000D_\ni)\tEvaluate whether there are topics relevant to AI\/ML BM OTA which RAN5 can contribute to RAN4 discussions. If so, list such areas and the RAN5 agreement status for each._x000D_\nii)\tIf there is agreement to i), then RAN5 needs to be added as secondary WG to the RAN4 WID which also may have to be updated with clear scope of RAN5 work. _x000D_\niii)\tIf there is no agreement to i), then re-visit the assessment at a later (TBD) meeting._x000D_\nObservation 2: RAN2 has been discussing and reached some key agreements in [2] wherein it was determined that \u2013_x000D_\n-\tSupported functionalities refer to functionalities that UE can indicate by using UE capability information (via RRC\/LPP signalling)._x000D_\n-\tRAN2 discuss further on the need of defining available functionalities separately from applicable functionalities._x000D_\nProposal 2: The UE capability information, once RAN2 signalling aspects and optional\/mandatory nature of the AI\/ML specific features on the UE are concluded, can be used for functionality identification and incorporated appropriately within the conformance test._x000D_\nObservation 3: There are significant testability issues ongoing discussion in RAN4 which will eventually impact RAN5 test procedures, MU element determination and MU value computation._x000D_\nObservation 4: In the case of Rel-15 FR2 OTA, it took more than 1 year from identification of Testability issues in RAN4 up to the point when RAN5 could trigger discussions_x000D_\n_x000D_\nObservation 5: TE vendors, who lead both RAN4 preliminary MU and RAN5 MU discussions had reported potential testability issues in RAN4, but preference\/focus was to complete performance requirements and wait till Perf part is complete and RAN5 can then assess MU. This took almost 1.5 years after Perf part completion_x000D_\n_x000D_\nProposal 3: Facilitate proactive involvement of RAN5 in preliminary MU\/testability assessment in RAN4 for AI\/ML related BM OTA testing, starting with the following broad steps:_x000D_\n?\tRequest RAN5 be added immediately as secondary WG with specific scope of collaborating\/contributing to preliminary and final MU assessment at appropriate trigger from RAN4._x000D_\n?\tAt a timepoint deemed suitable by RAN4 (for e.g close to finish of test methodology associated with any of the BM-related requirements), RAN4 triggers a request to RAN5 for collaboration and contributions on preliminary MU assessment.","secretary_remarks":"","agenda_item_sort_order":1220,"ainumber":"4.2.3","ainame":"AI\/ML Testability & Interoperability","tdoc_agenda_sort_order":75020,"status":"noted","reservation_date":"2024-08-11 21:59:50","uploaded":"2024-11-22 22:11:04","revisionof":"","revisedto":"","release":"","crspec":"","crspecversion":"","workitem":"","crnumber":"","crrevision":"","crcategory":"","tsg_crp":"","lsreplyto":"","lsto":"","Cc":"","lsoriginalls":"","lsreply":"","link":"https:\/\/www.3gpp.org\/ftp\/TSG_RAN\/WG5_Test_ex-T1\/TSGR5__105_Orlando\/Docs\/R5-247502.zip","group":"R5","meeting":"R5-105","year":2024,"uicc_affected":"","me_affected":"","ran_affected":"","cn_affected":"","clauses_affected":"","crsinpack":null,"crsinpacknumber":0},
{"name":"R5-247647","title":"Discussion on handling of AI\/ML in RAN5","source":"CMCC, AT&T, CAICT, Ericsson, Huawei, MediaTek, Nokia, ZTE","contact":"Ingbert Sigovich","contact-id":28887,"tdoctype":"discussion","for":"Endorsement","abstract":"","secretary_remarks":"","agenda_item_sort_order":1220,"ainumber":"4.2.3","ainame":"AI\/ML Testability & Interoperability","tdoc_agenda_sort_order":67071,"status":"noted","reservation_date":"2024-11-28 19:08:08","uploaded":"2024-11-28 19:11:06","revisionof":"R5-246707","revisedto":"","release":"","crspec":"","crspecversion":"","workitem":"","crnumber":"","crrevision":"","crcategory":"","tsg_crp":"","lsreplyto":"","lsto":"","Cc":"","lsoriginalls":"","lsreply":"","link":"https:\/\/www.3gpp.org\/ftp\/TSG_RAN\/WG5_Test_ex-T1\/TSGR5__105_Orlando\/Docs\/R5-247647.zip","group":"R5","meeting":"R5-105","year":2024,"uicc_affected":"","me_affected":"","ran_affected":"","cn_affected":"","clauses_affected":"","crsinpack":null,"crsinpacknumber":0}]