It goes back to Paul Karger's MULTICS Security Evaluation where he invented and described the attack. Thompson learned it from him. Karger invented a lot of attacks and security techniques a decade or more ahead of the hacking community.
The solution to this in the Orange Book (TCSEC) days in the 1980's-1990's was a system fully traceable from requirements to code, proven to embed a security policy, and analyzable and buildable from source locally by the customer using existing, trusted tools. Eventually, people added hashes for the code and data.
So, your program that combines source files or checks dependencies would be fully specified in its success and failure states. Only combinations of functions leading to a provably-secure state are even allowed. If you can't do that, the feature is too complex to allow. Human pentesters review it from design to algorithms to building it to spot ways attacks might happen.
That's what it takes to build software that usually resists subversion. Most software isn't built that way. It can't be because the priorities of developers and customers work against it. So, we'll continue to see clever attacks that exploit systems not designed to high security standards.
For this topic, I recommend David A. Wheeler's page on Software, Configuration Management Security because it covers many issues with it in mostly-centralized systems.
What happens when "security" includes a time component? A shocking number of modern systems depend on time, either in enough time having elapsed to prove something about the attacker, or in little enough time elapsing as a critical component of the system in question. That feels like it escapes the bounds of your definitions and is also somewhat unavoidable. Is that rectifiable somewhere?
> Most software isn't built that way. It can't be because the priorities of developers and customers work against it.
The most significant such priority may be the costs of paying developers and of time to delivery.
The dramatic reductions in those costs due to LLMs enable us to produce much more quantity and/or quality. Many complain now about quantity, so perhaps we are finally at a stage where we don't need much more software, and can focus on quality. Also, LLM attackers create demand for higher quality.
In other words, LLMs might enable us to some of these things that were impossible before.
It's very exciting to see the work on combining AI models with static analysis, test generation, formal proof, and refactoring. All of these suggests we might see high assurance (EAL6+) developed rapidly in the future. At least for combinations of well-understood concepts.
You don't mention generating code from scratch; is that on purpose? Perhaps a developer could add many constraints that would have been too expensive previously - e.g., code in Spark, use only these ___ system calls, never use this riskier capability, use only these ___ libraries, etc.
The LLM might take longer than otherwise, but still be fast enough. The new technology, like every useful one, changes the tradeoff equation.
Yeah, GPT4 was one-shotting utilities that GPT3 Davinci couldn't. So, I'd have my limited tokens on GPT4 crank out the initial program before iterating with my abundant, GPT3 tokens.
While I'm not a lawyer, the legal advice I've received on various topics include:
1. Most laws are made about humans. If it's AI, it's often treated like a tool the human is using. So, change "I did this with AI" to "I did this with (other tool here)." The case law on those situations might give hints to what will happen.
2. Intent matters. Did you intend to do damage?
3. If a tool might cause damage, but you didn't prevent that, then someone might claim negligence. There's a lot of legal articles about torts for damages due to negligence. I personally believe a lot of agent use should be considered negligent. By default, I don't connect them to the Internet or my whole filesystem because I know they might do unforeseen damage.
Those are the three that come to mind most in such cases. You'd have to ask a lawyer. There's another risk of even using a lawyer, though.
For using AI agents, you must consider civil and criminal law because its problems are spread across them. Most lawyers in my area do one or the other. You might have to pay two retainers at $5,000-$8000 each or one, expensive firm with combined expertise. Just knowing your legal risk with agents might cost more than they'd make or save you vs just using human-driven AI's.
I enjoyed the article. He has good points. I'll emphasize one and add a other.
1. CPU/memory performance on cheap or throwaway systems has a niche benefit. Mostly poor people. I'm still usually on an ancient Thinkpad with a 2nd gen i7. It runs native apps really fast to this day. Almost all GUI cuz I agree with the OP but my apps are TUI by default to keep them lean and fast.
2. Security. While I don't aim for it these days, it was much easier to make textual apps securely than GUI apps. Secure OS's from the 90's already secured console apps. TX, Nitpicker, and EROS made progress on GUI's but there's high complexity still. I'll note the OP's idea of GUI front ends is basically what we did to isolate the GUI part in a dedicated partition with messages it sends checked by the secure component.
Other than those observations, I'm with OP where I'm tired of TUI's if GUI's are that easy now. I considered trying it with some lightweight, cross-platform frameworks. Anybody tried some with cheap AI's?
My proposal was using actual curriculums to ensure that's all that's in there. Also, there could be a peformance boost if doing that first. We'd need funding to license or buy them.
Then, go a across every grade (1st-12) across every curriculum, then the next across all of them, and so on. Checkpoint it at each grade level. Also, see how many epochs we need per grade to soak up the material. Dedicated fine-tuning for each grade matched to its capabilities. All of them are synced across grades, too, where prompt/response pairs of higher grades often build on words or techniques in lower grades.
Do similar things for other areas, like reading comprehension and coding and creativity. Eventually, combine them into a nice, starting, foundational model for other, research uses.
Beware of copyright issues for API's and patent issues about reimplementations. Wolfram seems serious about his I.P.. After the Oracle case, I'm not reimplementing any language unless it's open with no patent trolling possible.
https://gwern.net/doc/cs/security/2002-karger.pdf
reply