AI辅助自动化Web漏洞探测：从信号提取到智能路由决策

暨 DRXAent开发日志 - 其一

一、背景：传统自动化扫描的局限

在渗透测试和SRC漏洞挖掘的日常工作中，传统的自动化扫描工具（如AWVS、Nessus）虽然能覆盖大量常见漏洞，但面临几个核心问题：

1. 上下文缺失：扫描器无法根据目标技术栈动态调整检测策略
2. 误报率高：基于静态规则的检测难以处理业务逻辑漏洞
3. 无法处理复杂链路：如需要先登录→获取Token→才能测试的IDOR漏洞
4. 缺乏反馈闭环：失败时不会分析原因并调整策略

基于DRXAgent项目的实践经验，本文分享一种AI辅助+自动化执行+智能路由的漏洞探测架构。

二、核心架构设计

2.1 整体闭环流程

这个设计的关键在于每一轮都会产生可消费的状态，而非简单的"执行-结束"模式。

2.2 信号驱动的动态路由

不同于传统扫描器的固定流程，我们的路由层基于运行时信号进行动态决策：

# app/skills/router.py 核心逻辑

def route(self, state: CybersecurityAgentState) -> tuple[SkillDSL | None, str | None, list[str]]:
    # 1. 检查是否有活跃的进行中技能
    active = self.registry.get(state.active_skill_id) if state.active_skill_id else None
    if active and self._next_experiment(active, state):
        return active, self._next_experiment(active, state), [active.skill_id]

    # 2. Web目标的首步必须是表面探测
    first_web_skill = self._first_web_surface_recon(state)
    if first_web_skill:
        return first_web_skill, experiment_id, [first_web_skill.skill_id]

    # 3. Web链式技能候选（基于证据推进）
    chained_skill = self._next_web_chain_skill(state)
    if chained_skill:
        return chained_skill, experiment_id, [chained_skill.skill_id]

    # 4. Agentic智能决策（多Agent规划）
    agentic_skill = self._agentic_next_step_skill(state)
    if agentic_skill:
        return agentic_skill, experiment_id, [agentic_skill.skill_id]

    # 5. 基于信号评分的候选排序
    scored = [(self._score(skill, state), skill) for skill in self.registry.all()]
    # LLM Tie-breaker：当分数接近时让LLM做最终决策
    selected = self._llm_tie_break(state, scored)

关键洞察：路由不是一次性决策，而是"条件触发→证据采集→再决策"的螺旋推进过程。

2.3 运行时上下文推导

在执行链中，每一步都会从证据中推导新的上下文键，供后续步骤消费：

# app/skills/executor.py 中的上下文推导

def _derive_context(self, state: CybersecurityAgentState) -> dict[str, Any]:
    context: dict[str, Any] = {}
    
    # 从历史证据中提取
    for ev in state.evidence:
        tool_data = ev.metadata.get("tool_data", {})
        
        # 提取JWT Token
        for token in tool_data.get("jwt_tokens", []) or []:
            if token and token not in jwt_tokens:
                jwt_tokens.append(token)
                
        # 从localStorage/sessionStorage提取凭证
        for area in ("local_storage", "session_storage"):
            for key, value in tool_data.get("storage", {}).get(area, {}).items():
                if re.search(r"(jwt|token|bearer|auth)", str(key), re.IGNORECASE):
                    jwt_tokens.append(str(value))
        
        # 提取表单中的文件上传字段
        for form in tool_data.get("forms", []) or []:
            for item in form.get("inputs", []) or []:
                if (item.get("type") or "").lower() == "file":
                    file_field_names.append(item.get("name"))
                    
        # 提取数字ID路径（IDOR候选）
        for item in tool_data.get("object_id_candidates", []) or []:
            if item and item not in object_id_urls:
                object_id_urls.append(item)
    
    return context

实践技巧：上下文推导函数是扩展探测能力的关键钩子。添加新的正则提取模式即可自动增强整个执行链。

三、Web技能链的实战示例

3.1 完整的授权测试链路

对于提供授权凭据的目标，系统会自动执行以下链路：

3.2 链式转换的亲和性评分

路由层通过_chain_transition_bonus方法为链式执行增加额外权重：

def _chain_transition_bonus(self, skill: SkillDSL, experiment_id: str, state: CybersecurityAgentState) -> float:
    # 获取当前聚焦的信号
    focused_signals = signal_names.intersection({
        "web-reflection", "sqli", "ssrf", "jwt", "upload-surface"
    })
    
    # 计算关键词匹配度
    keyword_hits = self._detector_focus_hits(skill, focus_text)
    if keyword_hits:
        bonus += min(6.0, 1.35 * keyword_hits)
        
    # 信号对齐奖励
    signal_hits = self._signal_alignment_hits(skill, signal_names)
    if signal_hits:
        bonus += min(8.0, 2.1 * signal_hits)
        
    # 聚焦信号技能匹配（如"sqli"信号匹配"br-web-sqli-probe"）
    focus_skill_hits = self._focused_signal_skill_hits(skill, focused_signals)
    if focus_skill_hits:
        bonus += min(9.0, 4.6 * focus_skill_hits)
        
    # 亲和性规则匹配
    bonus += self._apply_affinity_rules(skill, affinity)
    
    # 实验路由提示匹配
    bonus += self._apply_experiment_routing_hints(skill, experiment_id, affinity)
    
    return bonus

实战效果：当br-web-js-api-extract提取到包含/api/users/{id}的fetch调用时，系统会：

1. 生成web-route和js-api信号
2. 为br-web-route-discovery增加8分奖励
3. 为br-web-idor-bola-probe增加10分奖励
4. LLM tie-breaker确认后进入IDOR探测

3.3 浏览器运行时证据采集

# app/browser/extractor.py 中的表面提取

class BrowserSurfaceExtractor:
    def capture(self, url: str, timeout: int = 8) -> BrowserSnapshot:
        root = self.session.fetch(url, timeout=timeout)
        html = root.body.decode("utf-8", errors="replace")
        parser = _BrowserHTMLParser()
        parser.feed(html)
        
        # 同源链接去重
        links = self._dedupe(
            self._absolute_same_origin(base_url, base_url, item) 
            for item in parser.links
        )
        
        # 从inline JS提取fetch意图
        fetch_intents: list[BrowserFetchIntent] = []
        for body in parser.inline_scripts:
            fetch_intents.extend(self._extract_fetch_intents(body, base_url, base_url))
            
        # 从JS提取localStorage/sessionStorage
        storage = self._extract_storage(body)
        
        # 提取前端路由（React Router/Vue Router等）
        frontend_routes = self._extract_frontend_routes(body, base_url, base_url)
        
        # 识别登录候选路径
        login_candidates = [
            item for item in [*api_candidates, *frontend_routes]
            if re.search(r"(login|signin|auth|session|token|oauth|admin)", item, re.IGNORECASE)
        ]
        
        return BrowserSnapshot(...)

一个真实案例：在某次测试中，br-web-js-api-extract从Vue路由中发现了/admin/users/:id，从localStorage中提取到了JWT Token，随后br-web-jwt-probe发现该Token未验证签名，最终生成了越权访问管理后台的复现脚本。

四、失败处理与自适应调整

4.1 失败反射机制

当实验失败时，系统不会简单跳过，而是分析失败原因并调整策略：

# app/skills/reflector.py

def reflect(self, skill: SkillDSL, result: ExperimentResult, 
            verification: VerificationResult) -> ReflectionResult | None:
    
    # 基于失败类型生成纠正措施
    if reason == FailureReason.INPUT_CONSTRUCTION_ERROR:
        actions.extend([
            "Normalize generated parameters before retry.",
            "Prefer a simpler single-purpose probe before expanding patterns."
        ])
    elif reason == FailureReason.TOOL_FAILURE:
        actions.extend([
            "Verify tool availability and argument schema.",
            "Capture the failing tool call in trace before changing skill strategy."
        ])
    elif reason == FailureReason.TARGET_BEHAVIOR_MISMATCH:
        actions.extend([
            "Re-check the parameter context against live evidence.",
            "Switch to adjacent detector or collect a broader response sample."
        ])
        # 排除相同参数组合避免无限重试
        excluded.append(f"{skill.skill_id}:{result.experiment_id}:same_params")
    elif reason == FailureReason.VERIFICATION_INSUFFICIENT:
        actions.extend([
            "Add a stronger verifier or collect a second evidence source.",
            "Do not mark the branch complete from this evidence alone."
        ])
    
    # LLM增强：获取更精细的失败分析
    advice = self._llm_advice(skill, result, verification, reason, actions)

4.2 LLM辅助的决策增强

# 路由阶段的Tie-breaker决策
choice = self.llm.structured(
    system=(
        "You are the BR Red Agent skill router tie-breaker. "
        "Choose only from provided candidates. "
        "Prefer evidence-grounded, least-assumptive next experiments. "
        "Return JSON with selected_skill_id, selected_experiment_id, rationale, risk_notes."
    ),
    user=compact_json({
        "objective": state.objective,
        "signals": [signal.model_dump(mode="json") for signal in state.skill_signals],
        "memory": state.memory.model_dump(mode="json"),
        "knowledge_graph_summary": self.graph_summarizer.summarize(state.knowledge_graph),
        "candidates": [...]  # 高分候选技能
    }),
    schema=RouterChoice,
)

关键设计原则：

• LLM只参与决策，不直接操作目标
• 提供结构化上下文（知识图谱摘要、信号列表、执行历史）
• 输出被约束为JSON Schema，确保可解析
• 失败时自动回退到确定性逻辑

五、实战配置示例

5.1 启动授权Web测试

curl -s -X POST http://127.0.0.1:8088/campaigns/run \
  -H 'Content-Type: application/json' \
  -d '{
    "objective": "对授权网站进行安全研究，先做被动Web面探测",
    "target": {
      "target_url": "http://target.example.com/",
      "login_username": "testuser",
      "login_password": "testpass"
    },
    "authorized_scope": true,
    "max_loops": 5
  }'

5.2 日志分析任务

curl -s -X POST http://127.0.0.1:8088/campaigns/run \
  -H 'Content-Type: application/json' \
  -d '{
    "objective": "Investigate API gateway logs for BOLA and injection attempts",
    "target": {
      "file_path": "/path/to/api_gateway_logs.jsonl"
    },
    "max_loops": 3
  }'

5.3 环境变量配置

# LLM配置（用于智能决策）
BR_RED_LLM_BASE_URL=https://api.lkeap.cloud.tencent.com/coding/v3
BR_RED_LLM_API_KEY=your_key_here
BR_RED_LLM_MODEL=your_model_name
BR_RED_LLM_TIMEOUT_SECONDS=8

# 存储配置
BR_RED_SQLITE_PATH=./data/br_red_agent.sqlite3

# 生成脚本目录（避免热重载干扰）
BR_RED_GENERATED_SCRIPT_DIR=/tmp/br_red_scripts/

六、开发心得与最佳实践

6.1 关于自动化与AI的分工

6.2 可复现性与安全边界

1. 生成可执行脚本：每个高风险探测生成独立的Python脚本，可人工审计后执行
2. 策略门控：command_exec和python_exec默认只用于授权验证和生成的replay脚本
3. 凭据脱敏：login_password在trace中自动替换为***REDACTED***
4. 执行预算：限制工具调用次数、高风险操作次数、单实验重复次数

6.3 扩展新技能的模板

# SKILL.md 示例
---
skill_id: br-web-custom-probe
name: Custom Vulnerability Probe
description: >-
  Detect custom vulnerability pattern based on specific indicators.
detectors:
  - tags: [custom, api]
    keywords: ["vulnerable_pattern", "indicator"]
experiments:
  - experiment_id: probe_custom_vuln
    objective: Probe for custom vulnerability with bounded requests
    steps:
      - tool_name: http_probe
        arguments:
          url: "{target_url}"
          param: "{web_param_names}"
    success_followups:
      - br-web-custom-exploit
    failure_followups:
      - br-web-route-discovery
verifiers:
  - check_id: custom_indicator_present
    kind: regex
    target: evidence.content
    value: "custom_vuln_signal"
    weight: 1.0
confidence_policy:
  success_threshold: 0.8
  failure_threshold: 0.2
---

## When to Use

When the target shows indicators of custom vulnerability.

## Workflow

1. Identify candidate parameters from runtime evidence
2. Send bounded probe requests
3. Check response for vulnerability signals

## Verification

- Response contains `custom_vuln_signal`
- No false positive patterns present

七、总结与展望

现在项目探索了一条"确定性骨架+AI增强"的安全自动化路径：

1. 信号驱动：从运行时证据动态推导上下文，而非预设扫描模板
2. 智能路由：结合确定性评分与LLM决策，兼顾效率与灵活性
3. 链式执行：基于证据推进的自动链路，减少人工编排
4. 失败适应：分析失败原因并动态调整策略
5. 可审计：生成可复现的脚本和完整的执行轨迹

开发日志：2025-04-28