By Kyle Lewis, Chair, Sirtfi Exercise Planning Working Group
Estimated reading time: 5 minutes
In fall 2025, a grad student, Caroline Sweet, was recruited by the hacktivist group known as Stop All Robots. She helped them compromise the user registration process for a medical research team comparing the results of medical applications using two different large language models (LLMs) in an endeavor to leverage emerging AI technology. By the time she was done, they created a fake malicious account under the name of Real Diamond. They were also able to compromise the principal investigator, Dr Needsa Grant, and a number of the developers and researchers assisting him: Arty Fishal, Salmon Floyd, and others.
Through spear phishing and trojan horse malware, Stop All Robots eventually got enough access through research and developer accounts to erase evidence that the LLMs were hallucinating while at the same time increasing the LLM hallucination rates by tampering with the LLM models themselves in the hopes of creating a catastrophic failure after adoption that would convince people to never use AI again… They almost got away with it, but the InCommon federation universities and institutions supporting the research had something up their sleeve that Stop All Robots was not expecting: REFEDS Sirtfi.
The above was the fictional scenario that 13 university, federation, and government security teams came together to practice using REFEDS Security Incident Response Trust Framework for Federated Identity (Sirtfi) during InCommon’s fourth annual cyber security cooperation exercise held Nov. 17-21, 2025.
The Sirtfi is part of InCommon’s Baseline Expectations for federation members and provides federation identity providers (IdPs) and service providers (SPs) with a means to communicate and coordinate security incident responses… but do our teams know when and how to use it?
Addressing a Variety of Scenarios
This year’s fictional scenario provided the SP and IdP teams a variety of scenarios to address, which were all interconnected in a larger narrative. To facilitate the learning experience, the Sirtfi Exercise Planning Working Group (SEPWG) adopted a deliberate story-driven methodology of scenario development focused on using Sirtfi as a learning objective. Each SP and IdP had different types of incidents to respond to, all tied to a coherent narrative. Narrated incident scenarios included:
- Defaced websites
- Phishing and social engineering
- Insider threat
- A user account which couldn’t be traced to a real person
- Altered research data
As the story unfolded over the multiple days of the script, SPs and IdPs discovering indicators of compromise used Sirtfi to notify other affected security teams. SPs practiced taking in multiple inputs from multiple IdPs, managing the unfolding event, and using Sirtfi to notify further affected parties when applicable. This year, the InCommon Federation security contact brought the affected teams together to share cross-information.
Response from the participants was positive with a shared desire to do another exercise next year. During the feedback session, a continuing theme from last year was that this exercise helped strengthen their organizations’ internal ties between their security teams and identity and access management teams.
The Exercise Control Cell observed that teams are not used to communicating outside their organizations, and so even when using Sirtfi, there was a reluctance to share information. There were also teams who were new to Federation and Sirtfi. In both these cases, continued learning events such as this exercise would benefit the InCommon federation with increased participation by helping to raise practical awareness of Sirtfi and federation mindedness in security incident investigation and cooperation.
It was also noted that using TLP AMBER vs. TLP AMBER+STRICT markings on incident notifications sometimes prevented sharing critical information with other playing teams. The Cybersecurity and Infrastructure Security Agency (CISA) definitions for these markings are open to interpretation depending on who is involved. The SEWPG will be offering a recommendation to the InCommon Community Trust and Assurance Board (CTAB) that the federation provide some interpretation guidance for InCommon incident response for what TLP AMBER vs. AMBER+STRICT means.
Looking Ahead
The SEPWG plans to reconvene early next year and continue to grow and expand opportunities to practice intra- and inter-federation incident response.
Members of the SEPWG will be guided through the story-driven methodology and have an opportunity to contribute to ‘the cookbook.’ They will also be trained on how to run such an exercise and have an opportunity to be part of the Exercise Control Cell. Organizations contributing time to help on SEWPG will also get priority for their security teams to participate in the capstone tabletop exercise. The planned capstone event will be another distributed tabletop exercise in November 2026.
Kyle Lewis is the vice president of cybersecurity strategy at InCommon Catalyst RDCT, the Chair of the SEPWG, and a member of the InCommon CTAB.
About the Sirtfi Exercise Working Group
The Sirtfi Exercise Working Group prepares members of the InCommon Federation community to handle a federated security incident by conducting one or more tabletop exercises to simulate aspects of responding to the real thing. Exercises are aimed to be learning opportunities, increasing familiarity with and shared understanding of key concepts and practices in the Sirtfi framework. The Sirtfi Exercise Working Group is chartered by the InCommon Community Trust and Assurance Board. Additional information is available on the working group wiki.