`https://dev.to` — developer blogging platform. Everything useful is available via a public REST API with no auth required. No browser needed for any read task.
Scanned 5/27/2026
Install via CLI
Skills Directoryopenskills install yangchuansheng/browser-harness-rust# DEV Community (dev.to) — Data Extraction
`https://dev.to` — developer blogging platform. Everything useful is available via a public REST API with no auth required. No browser needed for any read task.
## Do this first
**Use the REST API — it returns clean JSON in ~150–250ms with no browser, no login, no JS rendering.**
```text
import json
articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&tag=python"))
# Each article: id, title, description, url, cover_image, tag_list, tags,
# published_at, published_timestamp, readable_publish_date,
# reading_time_minutes, positive_reactions_count,
# public_reactions_count, comments_count, user, organization,
# flare_tag, collection_id, slug, path, canonical_url,
# social_image, language, subforem_id
```
The API serves **V0 (beta) by default** and emits a `Warning: 299` header on every response. Suppress it silently with the V1 `Accept` header (same data, no deprecated warning):
```text
import json
import urllib.request, gzip
def dev_get(url):
h = {
"User-Agent": "Mozilla/5.0",
"Accept-Encoding": "gzip",
"Accept": "application/vnd.forem.api-v1+json",
}
with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=20) as r:
data = r.read()
if r.headers.get("Content-Encoding") == "gzip":
data = gzip.decompress(data)
return data.decode()
articles = json.loads(dev_get("https://dev.to/api/articles?per_page=10&tag=python"))
```
Or just use `http_get` directly if you don't care about the warning header noise.
---
## Common workflows
### Articles by tag
```text
import json
articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&tag=python"))
# Paginate with &page=2, &page=3 etc. (1-indexed)
for a in articles:
print(a['id'], a['positive_reactions_count'], a['title'][:60])
```
Confirmed working tags: `python`, `javascript`, `typescript`, `rust`, `go`, `webdev`, `tutorial`, `react`, `devops`, `ai`, `beginners`.
### Top articles by time window
```text
import json
# top=N means "top articles from the last N days"
top_day = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=1"))
top_week = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=7"))
top_month = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=30"))
top_year = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=365"))
```
### Articles by username
```text
import json
articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&username=ben"))
# Paginates cleanly: page=1, page=2 etc. Return distinct IDs, no overlap.
```
### New and rising articles
```text
import json
fresh = json.loads(http_get("https://dev.to/api/articles?per_page=10&state=fresh")) # very new
rising = json.loads(http_get("https://dev.to/api/articles?per_page=10&state=rising")) # gaining traction
# state=all returns 0 results (requires auth, not useful unauthenticated)
```
### Single article by ID (adds body_html and body_markdown)
```text
import json
article = json.loads(http_get("https://dev.to/api/articles/3442047"))
# Full article adds two fields not in list response:
# body_html — rendered HTML (safe to display directly)
# body_markdown — raw Markdown source
print(len(article['body_html']), len(article['body_markdown']))
```
### Single article by username/slug
```text
import json
# path field from list response is "/username/slug"
article = json.loads(http_get("https://dev.to/api/articles/ben/some-article-slug"))
```
### Tags — popular list with colors
```text
import json
tags = json.loads(http_get("https://dev.to/api/tags?per_page=10"))
# Fields: id, name, bg_color_hex, text_color_hex, short_summary
# Sorted by popularity. Paginate with &page=2 etc.
for t in tags:
print(t['name'], t['bg_color_hex'], t['text_color_hex'])
# e.g. webdev #562765 #ffffff
# javascript #f7df1e #000000
# ai #17fd1a #ffffff
```
### User profile
```text
import json
user = json.loads(http_get("https://dev.to/api/users/by_username?url=ben"))
# Fields: type_of, id, username, name, twitter_username, github_username,
# summary, location, website_url, joined_at, profile_image
print(user['id'], user['username'], user['summary'])
# e.g. 1 ben "A Canadian software developer who thinks he's funny."
```
`joined_at` is a human string like `"Dec 27, 2015"` — not ISO 8601. Parse with `datetime.strptime(user['joined_at'], "%b %d, %Y")`.
### Comments on an article
```text
import json
comments = json.loads(http_get("https://dev.to/api/comments?a_id=3442047"))
# Returns top-level comments only (replies nested under children key)
# Fields per comment: id_code (string, not int!), type_of, body_html,
# created_at, user (dict), children (list of same shape)
for c in comments:
print(c['id_code'], c['user']['username'], c['created_at'])
for reply in c.get('children', []):
print(" reply:", reply['id_code'], reply['user']['username'])
```
### Single comment by id_code
```text
import json
comment = json.loads(http_get("https://dev.to/api/comments/36lnc"))
# Same fields as above: id_code, body_html, created_at, user, children
```
### Bulk tag fetch (parallel)
```text
import json
from concurrent.futures import ThreadPoolExecutor
tags = ['python', 'javascript', 'typescript', 'rust', 'go',
'devops', 'webdev', 'tutorial', 'productivity', 'react']
def fetch_tag(tag):
data = json.loads(http_get(f"https://dev.to/api/articles?per_page=5&tag={tag}"))
return tag, data
with ThreadPoolExecutor(max_workers=3) as ex:
results = dict(ex.map(lambda t: fetch_tag(t), tags))
# 10 tags × 5 articles each: ~0.67s total with max_workers=3
```
---
## Endpoint reference
| Endpoint | Auth | Key params | Latency |
|----------|------|-----------|---------|
| `GET /api/articles` | None | `tag`, `username`, `top`, `state`, `page`, `per_page` | ~200ms |
| `GET /api/articles/{id}` | None | — | ~80ms |
| `GET /api/articles/{username}/{slug}` | None | — | ~200ms |
| `GET /api/tags` | None | `page`, `per_page` | ~190ms |
| `GET /api/users/by_username?url={username}` | None | — | ~190ms |
| `GET /api/comments?a_id={article_id}` | None | — | ~160ms |
| `GET /api/comments/{id_code}` | None | — | ~150ms |
| `GET /api/listings` | None | `category`, `page`, `per_page` | ~260ms (returns 0) |
**Listings endpoint returns 0 results.** The `/api/listings` endpoint is documented but returns an empty array for all categories (`jobs`, `forsale`, `education`, `cfp`) without auth. Skip it.
---
## Pagination
All list endpoints paginate with `page=` (1-indexed) and `per_page=`:
```text
import json
def get_all_articles_by_tag(tag, max_pages=5):
results = []
for page in range(1, max_pages + 1):
batch = json.loads(http_get(
f"https://dev.to/api/articles?per_page=30&tag={tag}&page={page}"
))
if not batch:
break
results.extend(batch)
return results
```
- `per_page` supports up to **1000** (confirmed). No documented max, but 1000 works in testing.
- No `total_count` field in list responses — you paginate until an empty array.
- Page ordering is consistent — confirmed no ID overlap between page 1 and page 2.
---
## Article field reference
All fields returned in list responses (single article adds `body_html` and `body_markdown`):
```
id int — article ID, stable, use for single-article fetch
title str
description str — auto-excerpt, never null
slug str — URL slug component
path str — "/username/slug"
url str — full canonical URL
canonical_url str — same as url for native posts; author's site URL for cross-posts
cover_image str|null — CDN URL or null (~30% of articles have no cover image)
social_image str — always present (generated if no cover_image)
tag_list list — e.g. ['python', 'ai', 'tutorial'] ← use this for code
tags str — same tags as comma-separated string "python, ai, tutorial"
published_at str — ISO 8601 UTC e.g. "2026-04-18T03:49:36Z"
published_timestamp str — identical to published_at
readable_publish_date str — human string e.g. "Apr 18"
reading_time_minutes int
positive_reactions_count int — hearts/likes count
public_reactions_count int — total reactions (usually same as positive_reactions_count)
comments_count int
user dict — name, username, twitter_username, github_username,
user_id, website_url, profile_image, profile_image_90
organization dict|null — present when posted under an org: name, username, slug,
profile_image, profile_image_90
flare_tag dict|null — {name, bg_color_hex, text_color_hex} — discussion/challenge badge
collection_id int|null — series/collection ID if part of a series
language str — e.g. "en"
subforem_id int|null
crossposted_at str|null — ISO datetime if cross-posted
edited_at str|null
last_comment_at str|null
created_at str — ISO 8601
type_of str — always "article"
```
---
## Rate limits
- **Burst limit: ~6 rapid sequential requests**, then HTTP 429.
- **Recovery: `Retry-After: 1` second** — wait 1s after a 429 and you're good again.
- No `X-RateLimit-*` headers in 200 responses — you only see `Retry-After` on the 429 itself.
- With `ThreadPoolExecutor(max_workers=3)`, 10 concurrent requests succeed without hitting the limit.
- No difference in limits between V0 (default) and V1 (`Accept` header) — same underlying rate limit.
- **No auth token tested** — all endpoints above work without `api_key`. Authenticated requests likely have higher limits.
Safe pattern for bulk fetching:
```text
import json, time
from concurrent.futures import ThreadPoolExecutor
def safe_fetch(url):
for attempt in range(3):
try:
return json.loads(http_get(url))
except Exception as e:
if '429' in str(e):
time.sleep(1) # Retry-After is 1s
continue
raise
return []
urls = [
f"https://dev.to/api/articles?per_page=10&tag={tag}"
for tag in ['python', 'javascript', 'typescript', 'rust']
]
with ThreadPoolExecutor(max_workers=3) as ex:
results = list(ex.map(safe_fetch, urls))
```
---
## Gotchas
- **`tag_list` (list) vs `tags` (string)** — both fields always present. `tag_list` is a Python list; `tags` is the same data as a comma-separated string. Use `tag_list` in code.
- **Comments have `id_code`, not `id`** — comment identifiers are alphanumeric strings like `"36lnc"`, not integers. The integer `id` field is absent from comment objects. Use `id_code` to fetch a specific comment via `GET /api/comments/{id_code}`.
- **Comments endpoint returns top-level only** — replies are nested under `children` recursively, not returned as a flat list. A thread with 100 total comments may only show 60 top-level objects; walk `children` recursively to count all.
- **`cover_image` can be null** — ~30% of articles have no cover image. Always guard: `a.get('cover_image') or a['social_image']` for a guaranteed image URL.
- **`flare_tag` is null for most articles** — only discussion/challenge posts carry it. It's a dict `{name, bg_color_hex, text_color_hex}` when present.
- **`published_at` == `published_timestamp`** — both fields contain identical ISO 8601 UTC strings. `readable_publish_date` is human-only (`"Apr 18"`, no year).
- **`joined_at` on user profile is not ISO** — it's `"Dec 27, 2015"`. Parse: `datetime.strptime(u['joined_at'], "%b %d, %Y")`.
- **`state=all` returns 0 results unauthenticated** — it's for the authenticated user's own feed. `state=fresh` and `state=rising` work without auth.
- **`top=N` means last N days** — `top=1` is last 24h, `top=7` is last week, `top=30` is last month, `top=365` is last year. Results differ from the `state=` param.
- **V0 warning header on every response** — `Warning: 299 - This endpoint is part of the V0 (beta) API…` appears on all responses without the `Accept` header. It's harmless but noisy. Suppress with `"Accept": "application/vnd.forem.api-v1+json"`.
- **No `total_count` in list responses** — paginate until an empty array. There is no way to know upfront how many total results exist.
- **Listings endpoint returns empty** — `GET /api/listings` and all category variants return `[]` without auth. Documented but non-functional publicly.
- **`/api/articles/{id}/comments` returns 404** — comments must be fetched via `GET /api/comments?a_id={id}`, not as a sub-resource of articles.
- **`canonical_url` may point off-site** — for cross-posted articles, `canonical_url` is the author's original blog URL, not dev.to. Use `url` for the dev.to link.
- **`organization` field is null for personal posts** — only present when the article was posted under an org account. Check before accessing sub-fields.
No comments yet. Be the first to comment!
