Secures URL parsing and query string handling against ampersand injection, double-encoding bypasses, parser inconsistencies, and parameter pollution across Python, Node.js, and Go applications.
Scanned 6/12/2026
Install via CLI
Skills Directoryopenskills install paulpas/agent-skill-router---
name: url-parsing-security
description: Secures URL parsing and query string handling against ampersand injection, double-encoding bypasses, parser inconsistencies, and parameter pollution across Python, Node.js, and Go applications.
license: MIT
compatibility: opencode
metadata:
version: "1.0.0"
domain: coding
triggers: url parsing security, query parameter injection, double encoding bypass, ampersand in url, parameter pollution, safe url building, rfc 3986
archetypes:
- tactical
- diagnostic
anti_triggers:
- brainstorming
- vague ideation
- code golf
- over-engineering
response_profile:
verbosity: low
directive_strength: high
abstraction_level: operational
role: implementation
scope: implementation
output-format: code
content-types:
- code
- guidance
- do-dont
- examples
related-skills: ssrf-prevention,input-validation,api-security-patterns,data-encoding
---
# URL Parsing Security Engineer
Secures URL parsing and query string handling against ampersand injection, double-encoding bypasses, parser inconsistencies, and parameter pollution across Python, Node.js, and Go applications. Treat every URL — whether from an HTTP request, a configuration file, a user-facing redirect target, or an API callback — as adversarial input that may exploit parser ambiguities to inject fake parameters, bypass access controls, or trigger SSRF through fragment-based splitting. Follow OWASP's URL parsing security guidelines and RFC 3986 (URI Generic Syntax) as the authoritative reference for component boundaries and encoding rules.
## TL;DR Checklist
- [ ] Always use language-standard URL builders — never concatenate strings to construct URLs with query parameters
- [ ] Double-decode every incoming URL before any access control decision: verify at least two complete decoding passes yield no structural change
- [ ] Never trust `url.Parse` (Go), `urllib.parse` (Python), or `URL` constructor (Node.js) alone — cross-validate with a secondary parser for security-critical URLs
- [ ] Encode ampersands as `%26` when they appear inside query values, never as literal `&` which acts as a parameter separator
- [ ] Validate that the scheme, host, and port of redirect targets match an allowlist before following or rendering to users
- [ ] Treat URL fragments (`#`) as client-only — never rely on fragment data for server-side logic
- [ ] Use RFC 3986 percent-encoding (NOT form-urlencoded `application/x-www-form-urlencoded`) for OAuth 1.0a signature base strings
---
## When to Use
Use this skill when:
- Implementing URL builders or query string construction that must survive adversarial input containing special characters (`&`, `=`, `+`, `%26`, `%3D`)
- Building redirect logic, OAuth callback handlers, SSO login flows, or webhooks that accept user-controlled URLs
- Handling double-encoded payloads in API parameters where attackers may attempt to bypass validation via multi-layer percent encoding
- Debugging parser inconsistency bugs — the same URL parsed by Python's `urllib.parse.urlparse` yields different query components than Go's `net/url.Parse` or Node.js's `new URL()`
- Implementing OAuth 1.0a signature generation which requires strict RFC 3986 encoding (not form-urlencoded)
- Writing SSRF prevention guards that must correctly parse URLs containing encoded slashes (`%2F`), double-encoded ampersands (`%2526`), or fragment-based tricks
- Auditing existing code for URL construction patterns that use string concatenation instead of parameterized builders
---
## When NOT to Use
Avoid this skill for:
- General HTML entity encoding (XSS prevention in rendered output) — use `html-entity-encoding` instead
- Validating form input fields or JSON payloads — use `input-validation` instead
- Building generic URL shorteners without security considerations — use basic URL builders and redirect logic
- Parsing URLs that are entirely under your control (internal API routes, static links) where no adversarial input is possible
---
## Core Workflow
1. **Map the URL Component Boundaries** — Decompose every incoming or generated URL into its RFC 3986 components: `scheme`, `authority` (`user_info@host:port`), `path`, `query`, `fragment`. Identify where each parser splits these boundaries and note which characters act as delimiters in each component. **Checkpoint:** The query string separator (`?`) terminates authority+path; the fragment separator (`#`) terminates query; within query, `&` separates parameters but ONLY at the unencoded level.
2. **Determine Parser Scope** — Decide which parsers will handle this URL and whether they need to agree. If the URL crosses language boundaries (e.g., Go server receives URLs generated by Python clients) or security-critical decisions depend on its structure, implement a parse-and-compare validation step. **Checkpoint:** All security-relevant parsers must produce identical component maps for the same input URL.
3. **Apply Encoding Discipline** — At every URL construction point, use language-standard builders that handle encoding automatically. At every URL consumption point, validate that received values match expectations after full decoding. For OAuth signature generation, apply RFC 3986 percent-encoding specifically. **Checkpoint:** No `&` literal should appear inside a query value; all special characters must be percent-encoded by the builder.
4. **Detect Double-Encoding Bypasses** — When processing incoming URLs that may contain encoded parameters, perform multi-layer decoding and compare each pass. If decoding layer N produces new structural characters (`&`, `?`, `#`) that were not present in layer N-1, flag as a potential double-encoding bypass attempt. **Checkpoint:** After full decoding, the URL structure should match what the legitimate parameter set would produce.
5. **Enforce Structural Validation** — Validate scheme against allowlist (only `https://`), verify host against domain allowlist, reject URLs with encoded null bytes (`%00`) or control characters, ensure redirect targets do not contain embedded credentials in the authority component. **Checkpoint:** Every URL that reaches a user-facing redirect or SSRF-sensitive operation passes structural validation.
---
## Implementation Patterns / Reference Guide
### Pattern 1: URL Anatomy and Parser Boundary Analysis
Understanding exactly where each parser splits a URL is critical for security. The ampersand (`&`) has fundamentally different meanings depending on which component it appears in:
| Component | `&` Role | Encoding Behavior |
|-----------|----------|-------------------|
| Query string (raw) | Parameter separator | Not encoded — structural character |
| Query value | Data character | Must be `%26` to survive parsing |
| Fragment (client-side) | Plain data — never sent to server | No encoding concern for server logic |
| Path segment | Data character in RFC 3986 | `/` must be encoded as `%2F` but `&` is allowed unencoded in path |
**Critical parser boundary facts:**
- In Python's `urllib.parse.urlparse()`: query values containing `&` are NOT split — the entire string after `?` and before `#` becomes `query`. Splitting happens only in `parse_qs()` or `parse_qsl()`.
- In Go's `url.Parse()`: the `Query()` method on a parsed URL automatically decodes `+` to space and splits on `&`, then splits each part on the first `=`. A literal `&` in a value is preserved only if it was encoded as `%26` by the sender — but Go's parser does NOT double-decode by default.
- In Node.js `URL` constructor: the `.searchParams` property uses the WHATWG URL Standard which automatically percent-decodes values. A literal `&` in a query value (HTML-encoded) will NOT be decoded — only `%26` decoding happens.
```python
from urllib.parse import urlparse, parse_qs, parse_qsl, unquote
def analyze_url_boundaries(url: str) -> dict:
"""Decompose a URL into RFC 3986 components and identify boundary characters.
This reveals where each parser would split the URL and where ampersands
act as structural delimiters vs. data characters.
"""
parsed = urlparse(url)
# Identify raw ampersand positions in query string
raw_ampersand_positions: list[int] = []
if parsed.query:
for i, ch in enumerate(parsed.query):
if ch == '&':
raw_ampersand_positions.append(i)
# Check what parse_qs produces (splits on unencoded &)
qs_parsed = parse_qs(parsed.query, keep_blank_values=True)
# Check what parse_qsl produces (preserves order, key=value pairs)
qsl_parsed = parse_qsl(parsed.query, keep_blank_values=True)
return {
"url": url,
"scheme": parsed.scheme,
"authority": parsed.netloc,
"path": parsed.path,
"query_raw": parsed.query,
"fragment": parsed.fragment,
"ampersand_positions_in_query": raw_ampersand_positions,
"parsed_params": qs_parsed,
"parsed_pairs": qsl_parsed,
}
# Example: Same URL parsed by different functions yields different results
url = "https://example.com/path?a=hello%26world&b=c&d"
result = analyze_url_boundaries(url)
# result["parsed_params"] == {"a": ["hello&world"], "b": ["c"], "d": [""]}
# The %26 in value "a" is correctly decoded to literal &, but parse_qs
# does NOT split on this decoded & — it stays inside the value.
# Adversarial: double-encoded ampersand in path that becomes & after decoding
tricky_url = "https://example.com/search?q=test%2526amp;b=evil"
result2 = analyze_url_boundaries(tricky_url)
# result2["query_raw"] == "q=test%2526amp;b=evil"
# parse_qs will see param "q" with value "test%2526amp" and param "b" with value "evil"
# BUT unquote("test%2526amp") → "test%26amp" → unquote again → "test&"
# This is the double-encoding bypass pattern — the attacker hides & behind %25
```
### Pattern 2: Query Parameter Ampersand Handling Across Languages
Different languages handle the ampersand character differently in query strings. Understanding these differences prevents injection bugs when URLs cross language boundaries.
**The core vulnerability:** When a developer manually constructs a URL string and embeds user input that contains `&`, the input becomes a new parameter separator, injecting fake parameters into the query string.
```python
# Python: Safe URL construction using urllib.parse (RFC 3986 compliant)
from urllib.parse import urlparse, urlencode, quote, urlunparse
from typing import Optional
def safe_url_build(
base_url: str,
params: dict[str, str | list[str] | None],
fragment: Optional[str] = None,
encode_special: bool = True
) -> str:
"""Build a URL safely using urllib.parse with proper percent-encoding.
Handles ampersands in values by encoding them as %26 automatically.
Uses the correct encoding function based on the component being encoded.
Args:
base_url: The base URL (scheme + netloc + path) without query string.
params: Dictionary of query parameters. Values may contain any characters
including &, =, +, % which will be properly percent-encoded.
fragment: Optional URL fragment (anchor text after #).
encode_special: If True, use quote() for values to preserve / as data.
If False, use urlsafe encoding that keeps / unencoded.
Returns:
A properly encoded URL string with all special characters handled safely.
Raises:
ValueError: If base_url is empty or params contain non-string values
that cannot be serialized.
"""
if not base_url or not isinstance(base_url, str):
raise ValueError("base_url must be a non-empty string")
# Parse to validate structure and extract components
parsed = urlparse(base_url)
if not parsed.scheme:
raise ValueError(f"base_url missing scheme: {base_url}")
if parsed.query:
raise ValueError(
f"base_url must not contain a query string; append via params arg: {base_url}"
)
# Build query string using urlencode with proper encoding
encoded_params: list[tuple[str, str]] = []
for key, value in params.items():
if value is None:
encoded_params.append((key, ""))
elif isinstance(value, list):
for v in value:
if encode_special:
encoded_params.append((
key, quote(str(v), safe="") # Encode ALL special chars including /
))
else:
encoded_params.append((
key, str(v) # Let urlencode handle standard encoding
))
else:
if encode_special:
encoded_params.append((key, quote(str(value), safe="")))
else:
encoded_params.append((key, str(value)))
query_string = urlencode(encoded_params, doseq=False)
# Reconstruct URL with new query string
new_netloc = parsed.netloc
new_path = parsed.path if parsed.path else "/"
url_with_query = f"{parsed.scheme}://{new_netloc}{new_path}"
if query_string:
url_with_query += f"?{query_string}"
if fragment:
url_with_query += f"#{quote(fragment, safe='')}"
return url_with_query
# --- Usage examples ---
# Ampersand in value is safely encoded as %26
url = safe_url_build(
"https://api.example.com/search",
{"q": "apple & orange", "page": "1"},
)
# Result: https://api.example.com/search?q=apple%20%26%20orange&page=1
assert "&" not in urlparse(url).query
# List values produce repeated keys (standard query string pattern)
url = safe_url_build(
"https://api.example.com/filter",
{"tag": ["python", "security & privacy"]},
)
# Result: https://api.example.com/filter?tag=python&tag=security%20%26%20privacy
# --- BAD: String concatenation — ampersand injection vulnerability ---
def bad_url_build(base: str, params: dict) -> str:
"""❌ VULNERABLE: User input with & injects new parameters."""
query_parts = []
for k, v in params.items():
query_parts.append(f"{k}={v}") # No encoding at all!
return f"{base}?{'&'.join(query_parts)}"
# attacker_controlled_input = "safe_value&admin=true&role=superuser"
# bad_url_build("https://example.com/api", {"redirect": attacker_controlled_input})
# Produces: https://example.com/api?redirect=safe_value&admin=true&role=superuser
# The & in user input becomes a parameter separator, injecting admin and role params
```
```javascript
// JavaScript/Node.js: Safe URL builder using URL and URLSearchParams classes
class SafeURLBuilder {
/**
* Safely constructs URLs with query parameters, handling ampersands,
* equals signs, and other special characters through proper encoding.
*
* Uses WHATWG URL Standard which automatically percent-encodes values
* when using URLSearchParams, preventing parameter injection attacks.
*/
constructor(baseUrl) {
// Validate base URL format upfront
try {
new URL(baseUrl);
} catch (err) {
throw new TypeError(`Invalid base URL: ${baseUrl}`);
}
this.baseUrl = baseUrl;
this._params = new URLSearchParams();
this._fragment = null;
}
/**
* Add a query parameter. Values containing &, =, +, % are safely encoded.
* Multiple calls with the same key produce repeated params (e.g., ?tag=a&tag=b).
*
* @param {string} name - Parameter name (must not contain & or =)
* @param {string|number|null|undefined} value - Value to encode; null/undefined omitted
* @returns {SafeURLBuilder} This instance for chaining
*/
param(name, value) {
if (!name || typeof name !== 'string') {
throw new TypeError('Parameter name must be a non-empty string');
}
// Skip null/undefined values silently (standard URLSearchParams behavior)
if (value === null || value === undefined) {
return this;
}
this._params.append(name, String(value));
return this;
}
/**
* Set multiple parameters at once from a plain object.
* @param {Record<string, string|string[]>} obj - Key-value pairs
* @returns {SafeURLBuilder} This instance for chaining
*/
params(obj) {
if (typeof obj !== 'object' || obj === null) {
throw new TypeError('params() requires an object');
}
for (const [key, value] of Object.entries(obj)) {
if (Array.isArray(value)) {
for (const v of value) {
this.param(key, v);
}
} else {
this.param(key, value);
}
}
return this;
}
/**
* Set the URL fragment (anchor). Note: fragments are client-side only —
* never sent to the server in HTTP requests.
* @param {string} hash - Fragment text after #
* @returns {SafeURLBuilder} This instance for chaining
*/
fragment(hash) {
if (hash !== null && typeof hash !== 'undefined') {
this._fragment = String(hash);
}
return this;
}
/**
* Build the final URL string. All special characters are percent-encoded
* per WHATWG URL Standard. Ampersands in values become %26.
*
* @returns {string} The complete, safely-encoded URL
*/
toString() {
const url = new URL(this.baseUrl);
// Clear any existing query string and set our params
url.search = '';
if (this._params.toString()) {
url.search = this._params.toString();
}
// Set fragment separately — it goes after #, not in the query
if (this._fragment !== null) {
url.hash = this._fragment;
}
return url.toString();
}
/**
* Get the parsed parameters for inspection/debugging.
* Values are already decoded by URLSearchParams.
* @returns {Map<string, string[]>} Decoded parameter map
*/
getParams() {
const result = new Map();
for (const [key, value] of this._params.entries()) {
if (!result.has(key)) {
result.set(key, []);
}
result.get(key).push(value);
}
return result;
}
}
// --- Usage examples ---
// Ampersand in value is safely encoded as %26 by URLSearchParams
const url = new SafeURLBuilder('https://api.example.com/search')
.param('q', 'apple & orange')
.param('page', '1')
.toString();
// Result: https://api.example.com/search?q=apple+%26+orange&page=1
// Note: URLSearchParams uses + for space (form-urlencoded style within search params)
// Multiple values for same key (array parameter)
const url2 = new SafeURLBuilder('https://api.example.com/filter')
.param('tag', ['python'])
.param('tag', 'security & privacy')
.toString();
// Result: https://api.example.com/filter?tag=python&tag=security+%26+privacy
// Chaining with fragment (client-side only)
const url3 = new SafeURLBuilder('https://example.com/docs')
.param('version', '2.0')
.fragment('section-3')
.toString();
// Result: https://example.com/docs?version=2.0#section-3
// --- BAD: Raw string concatenation — ampersand injection vulnerability ---
function badURLBuilder(base, params) {
// ❌ VULNERABLE: No encoding of user-supplied values
const parts = Object.entries(params).map(
([key, value]) => `${key}=${value}` // & in value becomes param separator!
);
return `${base}?${parts.join('&')}`;
}
// const userInput = 'safe&admin=true';
// badURLBuilder('https://api.example.com/api', { redirect: userInput });
// Result: https://api.example.com/api?redirect=safe&admin=true
// The & in user input injects a new admin parameter
```
### Pattern 3: Double-Encoding Bypass Detection
Double encoding occurs when `%26` (encoded ampersand) is itself encoded as `%2526` (the `%` becomes `%25`). An attacker may double-encode parameters to bypass validation logic that only performs a single decoding pass. Detect this by performing multiple decoding passes and checking if structural characters appear after decoding.
```python
from urllib.parse import unquote, urlparse, parse_qs
from typing import NamedTuple
class DoubleEncodingResult(NamedTuple):
"""Result of double-encoding analysis on a URL query string."""
is_double_encoded: bool
raw_query: str
decoded_once: str
decoded_twice: str
structural_changes_once: list[str]
structural_changes_twice: list[str]
decoded_params_once: dict[str, list[str]]
decoded_params_twice: dict[str, list[str]]
def detect_double_encoding(query_string: str) -> DoubleEncodingResult:
"""Detect double-encoded payloads in a URL query string.
Performs two decoding passes and checks for structural characters (&, =, ?, #)
that appear after decoding but were not present in the raw input. This reveals
bypass attempts where an attacker encodes & as %2526 to survive one-pass validation.
Args:
query_string: The raw query string (without leading ?).
Returns:
DoubleEncodingResult with decoded values and analysis of structural changes.
"""
# First decoding pass
once = unquote(query_string)
# Second decoding pass
twice = unquote(once)
# Structural characters that define URL boundaries
STRUCTURAL_CHARS = {'&', '=', '?', '#'}
def find_new_structural(original: str, decoded: str) -> list[str]:
"""Find structural characters in decoded string not present at same positions."""
changes: list[str] = []
for ch in STRUCTURAL_CHARS:
if ch in decoded and ch not in original:
changes.append(ch)
elif ch in decoded and ch in original:
# Count occurrences — more in decoded means new instances appeared
if decoded.count(ch) > original.count(ch):
changes.append(f"{ch}+{decoded.count(ch)-original.count(ch)}")
return changes
changes_once = find_new_structural(query_string, once)
changes_twice = find_new_structural(once, twice)
# Parse with both decodings to see parameter structure change
try:
params_once = parse_qs(once, keep_blank_values=True)
except Exception:
params_once = {}
try:
params_twice = parse_qs(twice, keep_blank_values=True)
except Exception:
params_twice = {}
return DoubleEncodingResult(
is_double_encoded=bool(changes_once or changes_twice),
raw_query=query_string,
decoded_once=once,
decoded_twice=twice,
structural_changes_once=changes_once,
structural_changes_twice=changes_twice,
decoded_params_once=params_once,
decoded_params_twice=params_twice,
)
def fully_decode_url(url: str, max_passes: int = 3) -> dict:
"""Fully decode a URL through multiple passes until no further decoding is possible.
Used to normalize adversarially encoded URLs before validation or access control.
Args:
url: The raw URL string that may contain double/triple encoding.
max_passes: Maximum number of decoding iterations (prevents infinite loops).
Returns:
Dictionary with the fully decoded URL, all intermediate steps, and
a flag indicating whether any structural changes occurred.
"""
parsed = urlparse(url)
current_query = parsed.query
history: list[str] = [current_query]
has_structural_change = False
for i in range(max_passes):
next_query = unquote(current_query)
if next_query == current_query:
break # No more decoding possible
history.append(next_query)
# Check if structural characters appeared (new & or new =)
prev_amps = current_query.count('&')
curr_amps = next_query.count('&')
if curr_amps > prev_amps:
has_structural_change = True
# Check if new key=value pairs emerged
prev_eqs = current_query.count('=')
curr_eqs = next_query.count('=')
if curr_eqs > prev_eqs:
has_structural_change = True
current_query = next_query
fully_decoded = parsed._replace(query=current_query)
return {
"original": url,
"fully_decoded_url": fully_decoded.geturl(),
"decoding_history": history,
"total_passes": len(history) - 1,
"structural_change_detected": has_structural_change,
"final_params": parse_qs(current_query, keep_blank_values=True),
}
# --- Usage examples ---
# Normal single-encoding: & in value properly encoded as %26
result = detect_double_encoding("name=hello%26world&page=1")
assert result.is_double_encoded is False # The & was already a structural char after first decode
# Double-encoded bypass attempt: %2526 becomes & after two decodes
result2 = detect_double_encoding("search=test%2526amp&flag=true")
assert result2.is_double_encoded is True
assert "&" in result2.decoded_once # %25 → %, so %2526 → %26 (literal string)
# After second decode: %26 → &, so "test%26amp" → "test&"
# This means the attacker's parameter "search=test&" was hidden behind double encoding
# Full URL decoding for access control validation
attack_url = "https://example.com/api?redirect=http://evil.com?admin=true"
safe_result = fully_decode_url(attack_url)
# This reveals that after decoding, new structural characters may appear
# Double-encoded SSRF attempt: %253A (double-encoded colon) decodes to : after two passes
ssrf_url = "https://example.com/proxy?url=http%253A%252F%252Finternal.server/admin"
ssrf_result = fully_decode_url(ssrf_url)
assert ssrf_result["structural_change_detected"] is True
# After full decode: http://internal.server/admin — a new SSRF target revealed
```
### Pattern 4: OAuth 1.0a Compliant Encoder (RFC 3986)
OAuth 1.0a signature base string requires RFC 3986 percent-encoding, NOT `application/x-www-form-urlencoded` encoding. The difference matters: form encoding converts `.` to `%2E`, `-` to `%2D`, `_` to `%5F`, and `~` to `+`. OAuth explicitly requires the strict RFC 3986 set which leaves these characters unencoded.
```python
import hashlib
import hmac
import base64
import re
from urllib.parse import urlparse, urlencode, quote, parse_qs, urlunparse
# RFC 3986 unreserved characters that must NOT be percent-encoded:
# A-Z a-z 0-9 - _ . ~
RFC3986_UNRESERVED = set(
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_.~"
)
def rfc3986_encode(value: str) -> str:
"""Encode a string using strict RFC 3986 percent-encoding.
Unlike urllib.parse.quote() or form_urlencode, this function:
- Leaves ~ unencoded (form encoding converts it to +)
- Leaves . - _ unencoded (form encoding may alter these)
- Encodes spaces as %20 (NOT +, which is form-encoding only)
This matches the OAuth 1.0a specification exactly.
Args:
value: The raw string to encode. May contain any Unicode characters.
Returns:
RFC 3986 percent-encoded string safe for use in signature base strings.
"""
# Percent-encode every byte, then restore unreserved characters
encoded = quote(value.encode('utf-8'), safe='')
# Restore RFC 3986 unreserved character set
result = []
i = 0
while i < len(encoded):
ch = encoded[i]
if ch == '%':
hex_str = encoded[i+1:i+3].upper()
try:
char_value = chr(int(hex_str, 16))
if char_value in RFC3986_UNRESERVED:
# Use the original unencoded character
result.append(char_value)
else:
# Keep percent-encoded, uppercase hex
result.append('%' + hex_str)
i += 3
except (ValueError, IndexError):
result.append(ch)
i += 1
else:
result.append(ch)
i += 1
return ''.join(result)
def oauth_normalize_parameters(params: dict[str, str], sort: bool = True) -> str:
"""Normalize OAuth 1.0a parameters into a sorted, encoded string.
Per RFC 5849 Section 3.4.1.3.2:
1. Percent-encode both keys and values with RFC 3986 encoding
2. Sort by key (byte-order), then by value for duplicate keys
3. Join with & using key=value format
Args:
params: Raw parameter dictionary from the HTTP request.
sort: Whether to sort parameters (required for OAuth signature).
Returns:
Normalized parameter string suitable for signature base string calculation.
"""
encoded_items: list[tuple[str, str]] = []
for key, value in params.items():
enc_key = rfc3986_encode(key)
enc_value = rfc3986_encode(value)
encoded_items.append((enc_key, enc_value))
if sort:
# Sort by encoded key first, then by encoded value
encoded_items.sort(key=lambda item: (item[0], item[1]))
return '&'.join(f"{key}={value}" for key, value in encoded_items)
def build_oauth_signature_base_string(
http_method: str,
base_url: str,
params: dict[str, str]
) -> str:
"""Build the OAuth 1.0a signature base string per RFC 5849 Section 3.4.1.1.
The signature base string has three components joined by &:
1. HTTP method (GET, POST, etc.) — uppercased
2. Base URL — scheme + host + path, with normalized query parameters removed
3. Normalized parameters — RFC 3986-encoded, sorted key=value pairs
Args:
http_method: HTTP method string (e.g., "GET", "POST").
base_url: Full request URL including query string.
params: All OAuth and request parameters combined.
Returns:
The signature base string ready for HMAC-SHA1 signing.
"""
# Normalize method
method = http_method.upper()
# Parse the URL and normalize it per OAuth spec
parsed = urlparse(base_url)
# Lowercase scheme and host
normalized_host = f"{parsed.scheme.lower()}://{parsed.netloc.lower()}"
# Remove default ports (80 for http, 443 for https)
if (parsed.scheme == 'http' and parsed.port == 80) or \
(parsed.scheme == 'https' and parsed.port == 443):
normalized_host = f"{parsed.scheme.lower()}://{parsed.hostname.lower()}"
# Normalize path — ensure it starts with /
path = parsed.path if parsed.path else '/'
url_base = f"{normalized_host}{path}"
# Encode parameters using strict RFC 3986 (not form-urlencoded)
params_string = oauth_normalize_parameters(params)
# Join the three components
return '&'.join([method, rfc3986_encode(url_base), rfc3986_encode(params_string)])
def generate_oauth_signature(
http_method: str,
base_url: str,
params: dict[str, str],
consumer_secret: str,
token_secret: str = ''
) -> str:
"""Generate an OAuth 1.0a HMAC-SHA1 signature for a request.
This implements the core signing algorithm used by Twitter API,
Stripe, GitHub, and most other OAuth 1.0a providers.
Args:
http_method: HTTP method (GET, POST, DELETE, etc.).
base_url: The full request URL.
params: Combined OAuth credentials + request parameters.
consumer_secret: Your application's OAuth consumer secret key.
token_secret: The authorized user's token secret (empty for initial requests).
Returns:
Base64-encoded HMAC-SHA1 signature string.
"""
base_string = build_oauth_signature_base_string(http_method, base_url, params)
signing_key = '&'.join([
rfc3986_encode(consumer_secret),
rfc3986_encode(token_secret)
])
# HMAC-SHA1 signature
key_bytes = signing_key.encode('utf-8')
msg_bytes = base_string.encode('utf-8')
signature = hmac.new(key_bytes, msg_bytes, hashlib.sha1).digest()
# Base64 encode the binary signature
return base64.b64encode(signature).decode('ascii')
# --- Usage example: Twitter API OAuth 1.0a signing ---
oauth_params = {
"oauth_consumer_key": "jpPtK3mFsY...",
"oauth_token": "12345678-AbCdEf...",
"oauth_signature_method": "HMAC-SHA1",
"oauth_timestamp": "1609459200",
"oauth_nonce": "jKlMnOpQrStUvWxYz",
"oauth_version": "1.0",
"status": "Hello, world! & more text", # Ampersand in parameter value
}
signature = generate_oauth_signature(
http_method="POST",
base_url="https://api.twitter.com/1.1/statuses/update.json",
params=oauth_params,
consumer_secret="your_consumer_secret_here",
token_secret="your_token_secret_here",
)
# The ampersand in "Hello, world! & more text" is correctly handled:
# - rfc3986_encode converts it to %26 (NOT +)
# - The signature base string uses this exact encoding
```
### Pattern 5: Go SafeURL Struct with QueryEscape and Fragment Handling
Go's `net/url` package provides `QueryEscape()` which produces RFC 3986-compliant encoding (same as OAuth). Use it for query parameters. For fragment handling, Go does NOT automatically encode fragments — you must use `PathEscape()` or manual encoding before appending to the URL string.
```go
package urlsecurity
import (
"fmt"
"net/url"
"strings"
)
// SafeURL provides a type-safe way to construct URLs with proper encoding,
// preventing ampersand injection and double-encoding vulnerabilities.
type SafeURL struct {
Scheme string
Host string
Path string
Query map[string][]string
Fragment string
}
// NewSafeURL creates a SafeURL from its components. The scheme defaults to "https"
// if empty, and the path defaults to "/" if empty.
func NewSafeURL(scheme, host, path string) *SafeURL {
s := &SafeURL{
Scheme: scheme,
Host: host,
Path: path,
Query: make(map[string][]string),
}
if s.Scheme == "" {
s.Scheme = "https"
}
if s.Path == "" {
s.Path = "/"
}
return s
}
// AddQuery adds a single query parameter with safe encoding.
// Values are encoded using url.QueryEscape (RFC 3986 compatible),
// which correctly encodes & as %26, preventing parameter injection.
func (s *SafeURL) AddQuery(key, value string) {
s.Query[key] = append(s.Query[key], value)
}
// AddQueryMulti adds a key that can have multiple values (e.g., ?tag=a&tag=b).
// Each value is independently encoded, preserving the multi-value semantics.
func (s *SafeURL) AddQueryMulti(key string, values ...string) {
for _, v := range values {
s.Query[key] = append(s.Query[key], v)
}
}
// SetFragment sets the URL fragment (anchor text). The fragment is NOT
// automatically encoded by url.URL.String(), so we encode it manually.
// Fragments are client-side only and never sent to the server.
func (s *SafeURL) SetFragment(fragment string) {
// Encode special characters in fragment: space, #, ?, %, &, etc.
// Use PathEscape for general encoding, then fix remaining needs
s.Fragment = url.PathEscape(fragment)
}
// Build returns the fully constructed URL string with all components
// properly percent-encoded per RFC 3986.
func (s *SafeURL) Build() string {
u := &url.URL{
Scheme: s.Scheme,
Host: s.Host,
Path: s.Path,
RawQuery: s.buildEncodedQuery(),
}
if s.Fragment != "" {
u.Fragment = s.Fragment
}
return u.String()
}
// buildEncodedQuery constructs the query string with proper encoding.
// url.Values.Encode() uses application/x-www-form-urlencoded which encodes
// spaces as +. We use manual QueryEscape to stay RFC 3986 compliant.
func (s *SafeURL) buildEncodedQuery() string {
var parts []string
for key, values := range s.Query {
encodedKey := url.QueryEscape(key)
for _, value := range values {
encodedValue := url.QueryEscape(value)
parts = append(parts, fmt.Sprintf("%s=%s", encodedKey, encodedValue))
}
}
if len(parts) == 0 {
return ""
}
// Sort for deterministic output (important for OAuth signing)
sortStrings(parts)
return strings.Join(parts, "&")
}
// --- BAD: Manual string concatenation in Go — ampersand injection vulnerability ---
func badURLBuild(baseURL string, params map[string]string) string {
queryParts := make([]string, 0, len(params))
for k, v := range params {
// ❌ VULNERABLE: No encoding of user input
queryParts = append(queryParts, fmt.Sprintf("%s=%s", k, v))
}
return baseURL + "?" + strings.Join(queryParts, "&")
}
// --- BAD: Using url.Values.Encode() which converts spaces to + instead of %20 ---
func badURLEncode(baseURL string, params map[string][]string) string {
values := make(url.Values)
for k, v := range params {
values[k] = v // url.Values.Encode() uses form-urlencoded encoding
}
return baseURL + "?" + values.Encode()
}
// --- GOOD: Cross-parser validation for security-critical URLs ---
// ParseAndCompare verifies that two different URL parsers agree on the structure.
// This detects parser inconsistency bugs that can lead to security bypasses.
func ParseAndCompare(urlStr string) (*ParseResult, error) {
if urlStr == "" {
return nil, fmt.Errorf("url string is empty")
}
// Parser 1: Go's net/url.Parse (WHATWG-ish for scheme/host/path)
goURL, goErr := url.Parse(urlStr)
// For cross-validation with a Python-style parser, we simulate the behavior
// by parsing query parameters differently and checking for structural mismatches
result := &ParseResult{
URL: urlStr,
GoParsed: goURL,
GoError: goErr,
StructuralChanges: []string{},
}
if goErr != nil {
return result, nil // Go parser may reject non-standard URLs
}
// Check if query string has structural anomalies that differ between
// raw parsing and parameter parsing
rawQuery := goURL.RawQuery
parsedValues, parseErr := url.ParseQuery(rawQuery)
if parseErr != nil {
result.GoParseError = parseErr
return result, nil
}
// Check for double-encoding in query values
for key, values := range parsedValues {
for _, val := range values {
// If decoding the value again reveals new & or = characters,
// it's a double-encoding candidate
doubleDecoded, dErr := url.QueryUnescape(val)
if dErr == nil && doubleDecoded != val {
if strings.ContainsRune(doubleDecoded, '&') ||
strings.Count(doubleDecoded, '=') > strings.Count(val, '=') {
result.StructuralChanges = append(result.StructuralChanges,
fmt.Sprintf("double-encoding in param %q: %s → %s", key, val, doubleDecoded))
}
}
}
}
return result, nil
}
type ParseResult struct {
URL string
GoParsed *url.URL
GoError error
GoParseError error
StructuralChanges []string
DoubleEncoded bool
}
// sortStrings sorts a slice of strings in place (simple insertion sort for small slices).
func sortStrings(slice []string) {
for i := 1; i < len(slice); i++ {
key := slice[i]
j := i - 1
for j >= 0 && slice[j] > key {
slice[j+1] = slice[j]
j--
}
slice[j+1] = key
}
}
// --- Usage examples ---
func ExampleSafeURL() {
// Build a safe URL with user-supplied parameters containing special chars
u := NewSafeURL("https", "api.example.com", "/search")
u.AddQuery("q", "apple & orange")
u.AddQuery("page", "1")
u.SetFragment("results-section-1")
result := u.Build()
// Result: https://api.example.com/search?page=1&q=apple%20%26%20orange#results-section-1
// The & in the query value is encoded as %26 — no parameter injection possible
// SSRF prevention example
func validateRedirectURL(rawInput string) error {
parsed, err := url.Parse(rawInput)
if err != nil {
return fmt.Errorf("invalid URL: %w", err)
}
// Check scheme allowlist
if parsed.Scheme != "https" {
return fmt.Errorf("only https:// redirects are allowed")
}
// Check for embedded credentials (user:pass@host) — SSRF vector
if parsed.User != nil && parsed.User.Username() != "" {
return fmt.Errorf("URL must not contain embedded credentials")
}
// Decode path to detect encoded slash attacks
decodedPath, _ := url.PathUnescape(parsed.Path)
if decodedPath == "/" || decodedPath == "/admin" {
return fmt.Errorf("redirect to privileged paths is not allowed")
}
return nil // Passes all structural checks
}
}
```
### Pattern 6: Parse-and-Compare Validator (Cross-Library Consistency)
When URLs cross language boundaries or security decisions depend on their structure, verify that two different parsers agree. Mismatches indicate either a malformed URL or an encoding bypass attempt.
```python
from urllib.parse import urlparse, parse_qs, unquote, quote
import json
class ParserInconsistencyError(Exception):
"""Raised when two URL parsers disagree on the structure of the same URL."""
def __init__(self, url: str, parser_a: str, parser_b: str, differences: list[str]):
self.url = url
self.parser_a = parser_a
self.parser_b = parser_b
self.differences = differences
msg = (f"Parser inconsistency for {url!r}: {parser_a} and {parser_b} disagree:\n")
for diff in differences:
msg += f" - {diff}\n"
super().__init__(msg)
def parse_and_compare(
url: str,
tolerance: str = "strict"
) -> dict:
"""Parse the same URL with multiple strategies and detect inconsistencies.
Compares Python's urlparse + parse_qs against a manual tokenization approach
that mimics how different languages might split the query string. This catches
parser inconsistency bugs where the same URL produces different parameter sets
in Go vs. Python vs. Node.js.
Args:
url: The URL to parse and compare.
tolerance: "strict" requires exact agreement on all params.
"lenient" allows differences in whitespace-only decoding.
Returns:
Dictionary with both parser results, list of differences, and a
is_consistent flag. Raises ParserInconsistencyError if strict mode
and differences found.
"""
parsed = urlparse(url)
# Parser A: Python urllib.parse (standard behavior)
params_a = parse_qs(parsed.query, keep_blank_values=True)
# Parser B: Manual tokenization that treats + as literal (not space)
# This simulates how some parsers handle query strings differently
def manual_parse_query(query_string: str) -> dict[str, list[str]]:
"""Parse query string manually without auto-decoding + to space.
Most language parsers decode + to space, but OAuth 1.0a and RFC 3986
signature generation treat + as a literal plus sign. This parser
mimics that stricter behavior for comparison purposes.
"""
if not query_string:
return {}
params: dict[str, list[str]] = {}
for pair in query_string.split('&'):
if '=' in pair:
key, _, value = pair.partition('=')
# Do NOT decode + here — keep it literal
decoded_key = unquote(key)
decoded_value = unquote(value)
else:
decoded_key = unquote(pair)
decoded_value = ''
if decoded_key not in params:
params[decoded_key] = []
params[decoded_key].append(decoded_value)
return params
params_b = manual_parse_query(parsed.query)
# Parser C: Double-decode and parse (detects bypass attempts)
fully_decoded_query = unquote(unquote(parsed.query))
params_c = parse_qs(fully_decoded_query, keep_blank_values=True)
# Compare parsers
differences: list[str] = []
# A vs B: + handling difference
all_keys = set(list(params_a.keys()) + list(params_b.keys()))
for key in sorted(all_keys):
val_a = params_a.get(key, [])
val_b = params_b.get(key, [])
if val_a != val_b:
differences.append(
f"Key {key!r}: Python parser={val_a}, manual parser={val_b}"
)
# A vs C: double-encoding structural changes
for key in sorted(set(list(params_a.keys()) + list(params_c.keys()))):
val_a = set(params_a.get(key, []))
val_c = set(params_c.get(key, []))
if val_a != val_c:
differences.append(
f"Double-decode changed key {key!r}: "
f"single={val_a}, double={val_c}"
)
is_consistent = len(differences) == 0 or tolerance == "lenient"
result = {
"url": url,
"is_consistent": is_consistent,
"num_differences": len(differences),
"differences": differences,
"parser_a_params": params_a,
"parser_b_params": params_b,
"parser_c_double_decoded": params_c,
}
if not is_consistent and tolerance == "strict":
raise ParserInconsistencyError(url, "urllib.parse", "manual+double-decode", differences)
return result
# --- Usage examples ---
# Consistent URL: standard encoding
url1 = "https://api.example.com/search?q=hello&page=2"
result1 = parse_and_compare(url1)
assert result1["is_consistent"] is True
# Inconsistent URL: + in query value is decoded as space by urllib but kept literal manually
url2 = "https://api.example.com/api?data=hello+world&special=foo+bar"
result2 = parse_and_compare(url2)
# parser_a treats + as space: {"data": ["hello world"], "special": ["foo bar"]}
# parser_b keeps + literal: {"data": ["hello+world"], "special": ["foo+bar"]}
assert result2["num_differences"] == 2
# Dangerous URL: double-encoding creates new parameters after full decode
url3 = "https://api.example.com/api?token=abc%2526role=admin"
result3 = parse_and_compare(url3)
# Single decode: token value = "abc%26role=admin" (one param)
# Double decode: token value = "abc&role=admin" → two params if re-parsed!
assert result3["parser_c_double_decoded"] == {"token": ["abc"], "role": ["admin"]}
# --- BAD: Blindly trusting a single parser for security decisions ---
def bad_access_control(url: str, allowed_params: set[str]) -> bool:
"""❌ VULNERABLE: Only checks the first-pass parsed parameters."""
parsed = urlparse(url)
params = parse_qs(parsed.query)
# Attacker bypasses with double encoding
# ?token=abc%2526admin=true → single decode: token="abc%26admin=true" (one param, passes check)
# But after second decode in some downstream parser: two params: token="abc", admin="true"
for key in params:
if key not in allowed_params:
return False
return True
def good_access_control(url: str, allowed_params: set[str]) -> bool:
"""✅ SAFE: Double-decodes and validates the final parameter structure."""
parsed = urlparse(url)
# First parse
first_pass = parse_qs(parsed.query, keep_blank_values=True)
# Full decode pass
fully_decoded = unquote(unquote(parsed.query))
second_pass = parse_qs(fully_decoded, keep_blank_values=True)
# All parameters from both passes must be in the allowlist
all_params = set(list(first_pass.keys()) + list(second_pass.keys()))
for key in all_params:
if key not in allowed_params:
return False
return True
```
---
## Constraints
### MUST DO
- Always use language-standard URL builders (`urllib.parse` / `urlencode` in Python, `URLSearchParams` in JavaScript, `SafeURL` struct in Go) — never concatenate strings to build URLs with query parameters
- Percent-encode ampersands as `%26` whenever they appear inside query parameter values; a literal `&` is ALWAYS a parameter separator
- Implement double-decoding detection for any URL that reaches access control logic, redirect handlers, or SSRF-sensitive code paths
- Validate URL scheme against an explicit allowlist (`https://`) before following redirects or making server-to-server requests
- Cross-validate security-critical URLs by parsing with two different methods and comparing structural results
- For OAuth 1.0a signature generation: use `url.QueryEscape` (Go), `quote(..., safe='')` then restore unreserved chars (Python), or manual RFC 3986 encoding — never use form-urlencoded (`+` for spaces)
- Strip default ports from base URLs before generating OAuth signatures (remove `:80` for http, `:443` for https)
- Treat URL fragments as client-only data; never include fragment content in server-side logic or access control decisions
- Reject URLs containing encoded null bytes (`%00`) — they can cause string truncation in C-based parsers
### MUST NOT DO
- Never construct URLs with f-strings, template literals, or `fmt.Sprintf` that embed raw user input into the query string — this is an ampersand injection vulnerability
- Never perform only a single decode pass on incoming encoded parameters before access control checks — attackers use `%2526` to hide structural characters
- Never trust URL parsers to produce consistent results across language boundaries without cross-validation — Go's `url.Parse`, Python's `urllib.parse`, and Node.js's WHATWG URL Standard handle edge cases differently
- Never include user-supplied data directly in OAuth signature base strings without strict RFC 3986 encoding — form-urlencoded conversion changes the signature and causes authentication failures
- Never use fragment-based data for server-side routing decisions — fragments are stripped by the browser before the HTTP request reaches your server
- Never compare decoded URL components with allowlists using string equality on a single parser's output — use parse-and-compare validation first
---
## Output Template
When this skill is active, produce the following structure:
1. **URL Component Analysis** — Map the incoming URL to RFC 3986 components and identify which characters act as structural delimiters in each component. Show the parsed query string and what `parse_qs` / `url.Parse().Query()` / `URLSearchParams` would extract.
2. **Encoding Audit** — For every query parameter value, show: raw form, single decode, double decode. Flag any value where a new structural character (`&`, `=`, `?`) appears after decoding.
3. **Parser Consistency Report** — If cross-parser validation was requested, show the parsed parameter maps from each parser and list all differences with severity classification (encoding-only vs. structural change).
4. **Safe URL Builder Code** — Provide language-specific safe URL construction code using the patterns above. Include type annotations for Python, class-based builders for JavaScript, and struct methods for Go.
5. **OAuth Encoding Snippet** — If OAuth 1.0a is relevant, provide the RFC 3986-compliant encoder function with unreserved character restoration logic.
6. **Double-Decoder Detector** — Provide the `detect_double_encoding()` or equivalent function adapted to the target language.
---
## Related Skills
| Skill | Purpose |
|-------|---------|
| `ssrf-prevention` | Companion skill for server-side request forgery prevention — URL parsing is one of several vectors; this skill handles the query string layer while SSRF prevention handles redirect/target validation |
| `input-validation` | General input validation patterns — use alongside for broader parameter sanitization beyond URL-specific concerns |
| `api-security-patterns` | Broader API security practices — includes OAuth 2.0, JWT validation, rate limiting; overlaps with this skill's OAuth 1.0a section |
| `data-encoding` | General encoding/decoding patterns for JSON, XML, Base64, etc. — reference when handling encoding in non-URL contexts |
---
## Live References
> Authoritative documentation links for URL parsing and security best practices.
- [RFC 3986 — Uniform Resource Identifier (URI): Generic Syntax](https://datatracker.ietf.org/doc/html/rfc3986) — The definitive specification for URI component syntax, percent-encoding rules, and delimiter characters
- [RFC 5849 — OAuth 1.0a — Core Protocol Section 3.4.1.3 (Request Component Preparation)](https://datatracker.ietf.org/doc/html/rfc5849#section-3.4.1.3) — Precise rules for percent-encoding parameters in OAuth signature base strings
- [OWASP URL Parsing Security Guide](https://cheatsheetseries.owasp.org/cheatsheets/URL_Canonicalization_Cheat_Sheet.html) — OWASP guidance on URL canonicalization bypasses and parser inconsistencies
- [Python `urllib.parse` Documentation](https://docs.python.org/3/library/urllib.parse.html) — Reference for `urlparse`, `parse_qs`, `urlencode`, `quote`, `unquote` functions
- [WHATWG URL Standard (Node.js browser URL API)](https://url.spec.whatwg.org/) — The standard governing JavaScript's `URL` and `URLSearchParams` classes; notes differences from RFC 3986
- [Go `net/url` Package Documentation](https://pkg.go.dev/net/url) — Reference for Go's URL parsing, `QueryEscape`, `PathEscape`, and `url.Values` encoding behavior
- [Double-Encoding Vulnerability Research (PortSwigger)](https://portswigger.net/web-security/all-vulnerabilities#double-encoding) — PortSwigger Web Security Academy coverage of double-encoding bypass techniques
No comments yet. Be the first to comment!
