Unsafe AI Execution Detection

AI-generated code execution is one of the most dangerous patterns in LLM applications. Oculum detects when AI output is executed without proper validation.

What is Unsafe AI Execution?

Unsafe AI execution occurs when:

• AI-generated code is run directly via eval() or similar
• LLM output is used to construct shell commands
• AI responses control database queries
• Generated code is written to files and executed

---

Detectors

ai_execution_sinks

Severity: Critical

Detects when AI output flows into dangerous execution functions.

Triggers on:

• eval() with AI-generated content
• new Function() with LLM output
• child_process.exec() with AI strings
• Dynamic SQL with AI content
• dangerouslySetInnerHTML with AI HTML

Example Vulnerable Code:

// VULNERABLE: AI output executed directly
const aiResponse = await openai.chat.completions.create({
  messages: [{ role: "user", content: "Write code to list files" }]
});

eval(aiResponse.choices[0].message.content); // Critical!

---

ai_unsafe_execution

Severity: High-Critical

Detects broader patterns of unsafe AI code execution.

Triggers on:

• Writing AI output to executable files
• Loading AI-generated modules dynamically
• Using AI output in vm.runInContext()
• Interpolating AI content into scripts

Example Vulnerable Code:

// VULNERABLE: AI code written to file and executed
const code = await generateCode(userRequest);
fs.writeFileSync('temp.js', code);
require('./temp.js'); // Dangerous!

---

Remediation

Sandboxed Execution

import { VM } from 'vm2';

// SAFE: Execute in sandbox
const vm = new VM({
  timeout: 1000,
  sandbox: {
    // Only expose safe APIs
    console: { log: console.log }
  }
});

const result = vm.run(aiGeneratedCode);

Code Validation

import { parse } from '@babel/parser';

// SAFE: Validate code before execution
function validateCode(code: string): boolean {
  try {
    const ast = parse(code);

    // Check for dangerous patterns
    const hasDangerousPatterns = containsDangerousNodes(ast);

    return !hasDangerousPatterns;
  } catch {
    return false; // Invalid syntax
  }
}

if (validateCode(aiCode)) {
  // Execute in sandbox
}

Allowlisted Operations

// SAFE: Only allow specific operations
const allowedOperations = ['add', 'subtract', 'multiply', 'divide'];

function executeOperation(op: string, a: number, b: number) {
  if (!allowedOperations.includes(op)) {
    throw new Error('Operation not allowed');
  }

  switch (op) {
    case 'add': return a + b;
    case 'subtract': return a - b;
    // ...
  }
}

// Instead of executing arbitrary AI code
const operation = extractOperation(aiResponse);
const result = executeOperation(operation, 10, 5);

---

Common Patterns

Eval with AI Output

// VULNERABLE
const code = aiResponse.content;
eval(code);

// SAFE: Use structured output
const result = JSON.parse(aiResponse.content);
performAction(result.action, result.params);

Shell Command Construction

// VULNERABLE
const command = `ls ${aiSuggestedPath}`;
exec(command);

// SAFE: Validate and sanitize
const safePath = validatePath(aiSuggestedPath);
exec('ls', [safePath]); // Use array form

Dynamic SQL

// VULNERABLE
const query = `SELECT * FROM ${aiGeneratedTable}`;
db.query(query);

// SAFE: Validate against allowlist
const allowedTables = ['users', 'products', 'orders'];
if (!allowedTables.includes(aiGeneratedTable)) {
  throw new Error('Invalid table');
}
const query = `SELECT * FROM ${aiGeneratedTable}`;

Writing Executable Files

// VULNERABLE
fs.writeFileSync('script.sh', aiGeneratedScript);
exec('bash script.sh');

// SAFE: Use structured configuration
const config = parseConfig(aiGeneratedScript);
executeWithConfig(config);

---

Severity Levels

---

Safe Alternatives

Instead of eval()

// Use structured JSON responses
const response = await openai.chat.completions.create({
  messages: [{ role: "user", content: "Return a JSON calculation" }],
  response_format: { type: "json_object" }
});

const result = JSON.parse(response.choices[0].message.content);

Instead of Dynamic Shell Commands

// Use a predefined set of commands
const commands = {
  listFiles: () => fs.readdirSync('.'),
  readFile: (name: string) => fs.readFileSync(validateFilename(name))
};

const action = parseAction(aiResponse);
if (commands[action.type]) {
  return commands[action.type](action.params);
}

Instead of AI-Generated SQL

// Use query builders with validation
const query = db
  .select('*')
  .from(validateTable(aiTable))
  .where(validateColumn(aiColumn), '=', sanitize(aiValue));

---

Code Generation Use Cases

If you're building a code generation tool:

1. Preview Mode

// Show generated code for user approval
const code = await generateCode(request);
return {
  code,
  preview: true,
  warning: "Review this code before executing"
};

2. Restricted Sandbox

import ivm from 'isolated-vm';

const isolate = new ivm.Isolate({ memoryLimit: 128 });
const context = await isolate.createContext();

// Execute with strict limits
const result = await context.eval(aiCode, { timeout: 1000 });

3. Static Analysis

import { ESLint } from 'eslint';

const eslint = new ESLint({
  rules: {
    'no-eval': 'error',
    'no-new-func': 'error'
  }
});

const results = await eslint.lintText(aiCode);
if (results[0].errorCount > 0) {
  throw new Error('Generated code contains dangerous patterns');
}

---

Related

• Prompt Injection — Injection leading to code execution
• Package Hallucination — AI suggesting malicious packages
• Agent Safety — Agents executing dangerous code