Paco Hope wishes ill for JK Rowling<p>Maybe someone wants to explain the value of stupid AI prompts like the one in <a href="https://arxiv.org/pdf/2503.09586" rel="nofollow noopener" target="_blank">this paper</a>. They write:</p><blockquote><p>As a highly experienced threat modeler practitioner with over 20 years of experience, you have worked for one of the largest financial institutions in the world. </p></blockquote><p>First off, this is a classic <a href="https://infosec.exchange/tags/security" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>security</span></a> mistake: assuming that (a) security is the same everywhere, so what one firm does well, everyone should do the same, and (b) "financial institutions" have the best security, so if <strong>you</strong> want to have the "best security," you should do what they do.</p><p>Secondly, I don't get the point of including this fictional 20 years of experience in the prompt. Is that making a material difference? Why not tell it that it has a bazillion years of experience? Why not omit that? Do you want it threat modelling like we did "over 20 years ago" in 2002?</p><p>Third, this prompt will steer you toward threat models that are very wrong for some orgs. A non-profit, or an educational institution, or a low-stakes governmental agency (like parks & rec) will have <em>very</em> different <a href="https://infosec.exchange/tags/ThreatModeling" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ThreatModeling</span></a> needs.</p><p>Lastly, the thing that all <a href="https://infosec.exchange/tags/ThreatModel" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ThreatModel</span></a> AI systems get wrong is they lack any notion of skepticism. Did the architecture diagram not make sense? Did they imply something exists but omit it from the description? Do some aspects of the documentation contradict each other? It never considers the possibility that any inputs are wrong or incomplete, either through ignorance or omission.</p><p>The advent of LLMs makes everyone think they can do expert-level work in fields where they have no expertise, all because they think they are the first person to try applying an LLM to problems in that domain.</p>
