AI has gotten a lot of attention for what it might mean for security, as well just about everything else. We were curious to see how an AI chatbot would handle processing public information about the security of WordPress plugins and if it would correctly warn that a plugin was known to be vulnerable. Our quick test involved Microsoft Copilot, which is accessible through Microsoft’s Bing search engine. We asked if the TablePress plugin was vulnerable. A web search could pull up our security scorecard for the plugin, which notes that, as of when it was checked in August, it was known to be vulnerable.
The results, which can be seen in full below, were interesting and not exactly surprising to anyone who takes a poor view of AI chatbots. In the four question conversation (that was a limit set by Microsoft), Copilot identified two different vulnerabilities as being the latest vulnerability in the plugin. It cited what doesn’t appear to be a reliable source for part of that. It also seems possible that the cited source is itself AI generated. The most problematic part of the response was the BS. It claimed a vulnerability had existed in version 2.3.1 of the plugin and been fixed in the next version, 2.3.2. Its next response claimed the vulnerability had existed in version 2.4.1 of the plugin and been fixed in the next version, 2.4.2. The cited source matched the first set of versions mentioned. [Read more]