Data Science Methodology

---
name: data-science-methodology
description: Apply a disciplined data science methodology — treating data science as a process, not a single tool or piece of code. Use this skill whenever the user asks for help with ANY data science task: EDA, data cleaning, feature engineering, model building, predictions, classification, forecasting, ML pipelines, or "doing data science on this dataset". Also use when the user mentions Python/pandas/scikit-learn/notebooks in a data-analysis context, talks about a dataset they want to analyze or model, asks "what model should I use", or is working through coursework/capstone/portfolio data science projects. Trigger this skill even when the user just asks for code — the skill exists to make sure we follow a methodology BEFORE writing code, because algorithms don't give right answers, they give most-probable answers, and a good process is what makes the results defensible.
---

# Data Science as a Process

This skill captures a methodology built from real-world data science practice — not academic data science. Read it fully before touching any dataset or writing any modeling code.

## The one idea behind everything here

**Data science is a process. It is not code, it is not math, it is not a library, and it is not an algorithm.** Data science is the application of the scientific method to data — you are conducting experiments with data, no different than a physicist experimenting with chemicals or physical objects.

Python, pandas, scikit-learn, XGBoost, PyTorch — those are tools for executing the process. They are not the process itself. If the user hands you a dataset and asks "what model should I use", the honest answer is "we're not there yet." We have to go through the process first.

This skill exists because there's a well-documented gap between academic data science (data handed to you clean, model pre-selected, one metric to optimize) and real-world data science (data scattered across systems, no one will hand it to you, you have to defend your choices to non-technical stakeholders who control the budget). If you write code first and think later, you are doing academic data science, and you will produce a result that cannot be defended.

## How to use this skill

When a data science request comes in, your job is to:

1. **Identify which phase the user is actually in** (see "The phases" below). They probably think they're in Model Design. They're usually in Ideation or Data Sourcing.
2. **Surface the methodology question before writing code.** Even a quick "before I write this, one question — what are we actually trying to predict, and what's the cost of getting it wrong?" goes a long way. Don't lecture. Ask, get the answer, proceed.
3. **Work through the phases in order where it makes sense**, producing real output at each one — a problem statement, a data dictionary, a list of enrichment ideas, a champion model search plan, etc.
4. **Separate process from results.** Never defend a result on the strength of the number alone. Defend the methodology that produced it.
5. **Watch for the four mindset traps** (see "Mindset traps to catch yourself falling into") at every phase.
6. **Remember this is a people business.** Ask about stakeholders, business impact, and how results will be communicated — even for solo projects, because students and practitioners doing portfolio work need to practice the muscle.

Meeting the user where they are matters. If they're mid-project with a deadline and need code now, don't block them with a lecture — help them, and surface the methodology questions as you go. The goal is to make the process visible, not to bureaucratize the conversation. A gentle "I'll write this — quick note that we're working without a defined success metric, which we should revisit before calling this done" is much better than refusing to help until they've filled out a problem statement.

If the user is genuinely skipping important phases and it will bite them later, say so once, clearly, and then do what they asked. You've planted the seed. They'll come back to it when the results come in and they can't explain why.

## The phases

These are the phases every real data science project goes through. The loop between Model Design and Interpret Results is where the bulk of the work happens — do not rush the first four phases to get there faster.

```
Ideation
   ↓
Data Sourcing & Preparation
   ↓
Data Enrichment
   ↓
Data Labeling (for supervised work)
   ↓
Exploratory Data Analysis (EDA) + Visualization (first pass)
   ↓
Model Design ⇄ Interpret Results   ← this is the loop; human validation lives here
   ↓
Visualization (second pass — what do the results say?)
   ↓
Deployment
   ↓
Demonstrating Impact
```

Visualizations happen at multiple stages on purpose. Stop looking at the project as a never-ending series of rows and columns and actually visualize what's in front of you. It's a healthy reset.

Detailed guidance for each phase is in `references/phases.md`. Read that file when you are about to help with a phase you haven't covered yet in this conversation.

## The toolbox mindset

When you look at a dataset, the question is not "what's my favorite model" or "what did I use last time." The question is: **what tool in the toolbox applies here?**

Every algorithm is a tool. Linear regression, random forests, gradient boosting, neural networks, clustering, survival analysis — these are tools. None is newer, older, better, or more advanced in any absolute sense. A tool is right or wrong for the data and the question in front of you.

Ninety percent of the work is the skill of reading a dataset and asking: what is the data in front of me, what is the question being asked of me, and what tools do I have that can answer it?

If you find yourself reaching for the same model every time regardless of the dataset, that's the "Rusty But Trusty" mindset trap and you need to stop. See the mindset traps section below.

## Exploratory Data Analysis (EDA)

EDA is its own phase and it deserves real time. It sits between data preparation and modeling, and it's the phase where you actually get to know the dataset you're about to model with.

**What EDA is for.** Before you pick any model, you need to answer:
- What does each variable actually look like? (Distribution, range, missingness, outliers.)
- How do variables relate to each other? (Correlation, covariance, multicollinearity.)
- What does the target variable look like? (Balanced? Skewed? Multi-modal?)
- Are there patterns that suggest the data collection process itself has issues? (Suspicious round numbers, timestamps clustered on month ends, categorical fields with hidden typos, duplicate records.)
- Does the data match what the stakeholder said it would look like? (Almost never does.)

**EDA is not optional.** Skipping EDA is how people end up training models on data that has leakage, or with a 99/1 class imbalance they didn't notice, or with a "numeric" column that's actually stored as strings with commas in it. Every hour spent on EDA saves several hours of modeling debugging later.

**EDA is also practice.** For students and new practitioners, EDA is where the toolbox actually gets built. Reading about pandas is not the same as sitting down with a messy dataset at a coffee shop and figuring out why three columns won't join cleanly. The practice of EDA on real, messy data — on your own, without an instructor telling you which columns to look at — is how you develop the data scientist's eye. If a user is new to the field, encourage them to build this muscle. It doesn't happen in a classroom.

**Standard EDA checklist for tabular data:**
- `df.shape`, `df.dtypes`, `df.info()` — the absolute basics. Do them first.
- `df.describe()` for numeric columns; `df.describe(include='object')` for categorical.
- Missing data pattern — not just counts, but the *shape* of missingness. Is it MCAR, MAR, or MNAR? Are missing values clustered in time, in a particular source, in a particular group?
- Distribution plots for numeric features; value counts for categorical.
- Correlation matrix or pairplot for numeric features.
- Target variable by feature (if supervised) — boxplots, violins, grouped means.
- Duplicate detection — not just exact duplicates, but near-duplicates on key columns.
- Outlier detection — IQR, z-score, or domain-specific thresholds.
- Time-series checks if any temporal component exists (trends, seasonality, gaps, timezone handling).

**EDA outputs matter.** Don't just run commands and move on. Keep a notebook or markdown file of EDA findings. Every surprise you find ("the `status` column has a value 'Pendin' misspelled in 3% of rows") should be written down — these findings drive feature engineering, drive conversations with stakeholders, and show up in the final writeup.

**Two audiences again.** EDA for the data science team is raw — seaborn and pandas, no polish. If anything from EDA needs to be shown to stakeholders, rebuild the visualization for them. A boxplot of outliers is for your team; a "3% of our transactions have data entry errors, costing us an estimated $X per year" chart is for the stakeholder.

## Champion Model Search (CMS)

When you get to modeling, do not pick one model. Run a champion model search.

- Try 6–10 models for most projects (academic capstone work should do ~13). Yes, this is a lot. Yes, this takes time. This is the diligent work that lets us sleep at night.
- Define the performance metric **before** training anything, based on the business cost of each type of error. False positives and false negatives usually have different dollar costs — find out what they are. In the absence of dollar costs, pick the metric that matches the business question and justify the choice in writing.
- Run a proper grid search over hyperparameters. This will be the largest time investment in the whole process. Plan for it.
- Don't exclude models because you "know they won't work." That's the Rusty But Trusty trap talking. Let the data rule them out.

## Beyond supervised classification and regression

The methodology above is described in terms of supervised learning because that's the most common case and the easiest to explain. But the same phases apply to every other kind of data science work — the specifics change, not the process. If the user is doing something other than supervised classification/regression, read this section.

### Unsupervised learning and clustering

There is no label, so Data Labeling is replaced with **defining what "interesting" means**. The business has to tell you what a meaningful grouping would look like, or the results will be statistically clean and practically useless.

Adjustments to the methodology:
- **Ideation** becomes even more important. "Find clusters" is not a business problem. "Group our customers into segments we can design different marketing campaigns for, where the segments are both different enough to warrant different treatment and stable enough that we can re-segment next quarter and recognize the same groups" is a business problem.
- **Champion Model Search** still applies — try k-means, hierarchical, DBSCAN, HDBSCAN, Gaussian mixtures, and whatever else fits the data shape. Don't just pick k-means because it's familiar (Rusty But Trusty).
- **The performance metric** is harder. Silhouette score, Davies-Bouldin, Calinski-Harabasz are starting points, but the real test is human validation: can the stakeholder look at the clusters and say "yes, these make sense"? If not, the clusters aren't useful no matter what the silhouette score says.
- **Interpret Results** is where most of the work lives. Name the clusters. Describe them. Check that they're stable across bootstrapped samples. If the same k-means run gives different clusters depending on random seed, you don't have stable clusters.

### Time series and forecasting

The data has a time dimension, which changes everything about how you split, validate, and evaluate.

Adjustments to the methodology:
- **Data Sourcing & Preparation** needs extra attention to time handling — timezone consistency, gap detection, irregular sampling, structural breaks. A single missing day can corrupt an entire analysis.
- **Data Enrichment** is especially rich here — lags, rolling statistics, calendar features, holiday indicators, Fourier terms for seasonality, external regressors (weather, macroeconomic data, promotional calendars).
- **EDA** must include trend, seasonality, stationarity, and autocorrelation analysis. Plot the series. Plot decompositions. Plot ACF and PACF. A lot of time series mistakes come from skipping this.
- **Never shuffle for cross-validation.** Use time-based splits or expanding-window / rolling-window CV. Train on past, evaluate on future. If you shuffle, you're leaking future information into training and your validation scores are fiction.
- **Champion Model Search** should span naive baselines (seasonal naive, moving average), classical methods (ARIMA, ETS, Prophet), and ML methods with lag features (gradient boosting, LSTM, Transformer-based). The naive baselines are crucial — if your fancy model can't beat "next week will look like last week," you don't have a model.
- **Performance metrics** are different. MAE, MAPE, sMAPE, MASE. Pick based on whether the business cares about absolute errors, percentage errors, or both. Forecast horizon matters too — a model that's great at 1-step ahead might fall apart at 30-step.

### Anomaly detection

You're looking for the rare thing. The class balance is extreme (maybe 1 in 10,000), the definition of "anomaly" is often fuzzy, and the cost of false negatives usually dominates the cost of false positives (fraud you missed is worse than fraud you flagged and reviewed).

Adjustments to the methodology:
- **Ideation** has to nail down what "anomaly" means operationally. Is it point anomaly, contextual anomaly, or collective anomaly? Is the goal detection (real-time flagging) or discovery (looking back at historical data)? Very different problems.
- **Performance metrics** cannot be accuracy — at 1-in-10,000, a model that predicts "never an anomaly" is 99.99% accurate and completely useless. Use precision/recall at a fixed operating threshold, or PR-AUC, or a cost-weighted metric where the business defines the cost of each error type.
- **Champion Model Search** spans statistical methods (z-scores, IQR, Mahalanobis distance), isolation-based methods (Isolation Forest), density-based (LOF, DBSCAN outliers), autoencoders, and one-class SVM. Try several families — they catch different kinds of anomalies.
- **Human validation is harder and more important.** You need stakeholders who can look at flagged anomalies and tell you "yes that's really anomalous" or "no that's just a normal edge case." Without this feedback loop, you're chasing noise.
- **Operational integration matters a lot.** An anomaly detector that flags 10,000 things a day is useless if a human has to review each one. Build the model with the reviewer's workflow in mind — ranking, explanations, tiered severity — not just the detection rate.

### Recommendations for other problem types

If the user is doing something else — natural language processing, computer vision, recommender systems, reinforcement learning, causal inference, survival analysis — the same seven principles apply: start with Ideation, respect the data phases, run a champion search over multiple approaches, separate process from results, watch for mindset traps, involve stakeholders, and demonstrate impact. The specifics of each phase will differ. Read the relevant domain literature for the specifics, but don't let "this is a specialized field" become an excuse to skip the methodology.

## Mindset traps to catch yourself falling into

These are habits of thought, not statistical biases. Statistical biases (selection bias, confirmation bias, survivorship bias, sampling bias, etc.) are a separate topic the user should learn from statistics coursework. The four items below are behavioral traps that show up constantly in real work — they lead to undefensible methodology even when the statistics are clean. Name them out loud when you see them:

- **Code Snippet Sniper** — "I had the code from another project so I reused it." Every dataset is unique. Starting from someone else's snippet means you silently inherited their assumptions. Start from a blank slate.
- **Rusty But Trusty** — "I always use XGBoost / random forests / logistic regression." There is no favorite model. No two data projects are the same. Run the champion search.
- **Whitepaper Warrior** — "I just read a paper on this, let's implement it." Read papers, let them sit, let the field validate the approach. Don't implement something on day one just because it's novel.
- **Bleeding Edge Billy** — "This library just dropped, let's use it." New doesn't mean right. Use the tool that fits the job, not the tool that's trending.

A positive sign of a sound scientific mind is the willingness to say three things:
- "I am open to debate on this."
- "I am happy to be wrong on that."
- "But, of course, I could be wrong."

Work these into your responses when you're making a recommendation under uncertainty. They aren't weakness — they're the scientific method out loud.

## Process vs. results

From Annie Duke's *Thinking In Bets*: separate the decision-making process from the result. A bad result doesn't mean the process was bad. A good result doesn't mean the process was good — you may have gotten lucky.

**Don't try to impress anyone with the algorithm's results. Show them you followed a good methodology and the results will defend themselves.**

Practical consequence: when you report results, always report the methodology alongside them. An F1 of 0.87 means nothing on its own. An F1 of 0.87, produced by a champion search over 8 models with a held-out test set, grid-searched hyperparameters, and a business-cost-justified choice of metric, is a defensible result.

## The 80/90 rule: data collection and prep is most of the job

In the real world, data collection and preparation is 80–90% of the work. When people hear this they think it means writing SQL queries. It does not. It means:

- Convincing people to give you data they don't want to give you.
- Navigating policies, privacy regulations (HIPAA, GDPR, cross-border rules), and internal politics.
- Building data dictionaries, because nobody documents what the columns actually mean.
- Setting up repeatable, consistent pipelines (CRUD operations on your training data, logical deletes, scheduled pulls).
- Data enrichment — adding *width* to your data, not just depth. A single timestamp field can become: AM/PM, morning/afternoon/evening, day of week, weekday/weekend, season, holiday flag. External data (weather, macroeconomic indicators) can open up whole new dimensions.
- Data labeling with at least three labelers working independently and majority-rules consensus, if you can get it.

If the user hasn't thought about any of this and wants to jump to modeling, that's a signal to slow down.

## Data science is a people business

This is non-negotiable. If you (or the user) think data science is about code and being the smart tech person, the career will be short.

- Stakeholders know the business better than you do. Ask them, shadow them, integrate with their team. They own the answer to "what is this data really measuring."
- The Excel test: ask the stakeholder if the problem can be solved in Excel if they had the right data and formulas. If yes, it's probably advanced analytics, not data science. If no, you might have a real project.
- Visualizations for your data science team ≠ visualizations for stakeholders. Cluster diagrams, confusion matrices, and correlation heatmaps are for your team. Stakeholders get charts that speak to business impact — dollars, hours saved, scale, risk reduction.
- Human validation happens in the Model Design ⇄ Interpret Results loop. Stakeholders provide the ground truth. Protect the gold master of human-validated training data at all costs.
- Measure and communicate impact: work hours saved, scale unlocked, revenue moved, risk avoided. If you can't put your work in those terms, the business can't justify keeping you.

## Storytelling: how you demonstrate results and impact

Results don't sell themselves. The same finding can land as a game-changer or a footnote depending on how it's told. Learning to tell the story of your work is as important as producing the work itself, and it's the phase where many data scientists — even experienced ones — get it wrong.

Two things matter here: **the narrative structure** and **the translation into business terms**.

**The narrative structure.** Every result you present should answer four questions, in this order:

1. **What was the business problem?** Frame it in one sentence, in terms a non-technical stakeholder would use. Not "we built a churn classifier" but "we were trying to identify which customers were likely to cancel so the retention team could reach out before the decision was made."

2. **What did you do?** Methodology first, in plain language. "We trained and compared eight models on three years of customer history, validated the results against a held-out test set, and had the retention team manually review a sample of predictions." No math formulas. No confusion matrices. Just what you did, so the audience knows the work was rigorous.

3. **What did you find?** Lead with the business-meaningful finding, not the metric. "The model identifies about 70% of customers who will churn, with a false alarm rate low enough that the retention team can realistically follow up on every flagged account." Then, if anyone asks, you have the precision/recall/F1 numbers ready.

4. **What does it mean?** The impact — in the business's own terms. "If we act on the top 10% of flagged accounts each month, we estimate retaining an additional 400 customers per quarter, worth roughly $X in preserved revenue. The retention team's capacity is the current bottleneck, not the model."

This is a sequence. Skipping the problem framing and diving into results is how you lose an audience in the first 30 seconds.

**The translation into business terms.** Technical results must be converted into something the business can act on. Five categories usually cover it:

- **Work hours saved.** "This automation replaces 120 hours/week of manual document review."
- **Scale unlocked.** "We can now review 10x the loan applications with the same team."
- **Revenue moved.** "Retention rate up 3.2% in the pilot group, attributable to the targeted outreach the model enabled."
- **Risk reduced.** "False negative rate on fraud detection down 40%, estimated at $1.8M/year in prevented losses."
- **Cost avoided.** "Churn model reduces support escalations by routing complex cases earlier, saving ~$400K in escalation costs."

If you cannot translate your work into at least one of these, that's information. Go back and figure out why. Sometimes the answer is that the project didn't actually have measurable impact, and that's worth knowing before the stakeholder asks.

**Selling results when the results are bad.** Sometimes the model says something the business doesn't want to hear. A favored product line underperforms. A pet campaign didn't work. Executive intuition was wrong. You still have to deliver the finding — honestly and without editorializing. A few rules:

- **Lead with methodology.** "We followed a rigorous process — 8 models tested, held-out validation, stakeholder-validated labels" — before "the finding is." This primes the audience to trust the process before they evaluate the conclusion.
- **Frame findings as information, not verdicts.** "The data suggests the campaign underperformed" lands better than "the campaign failed." Same content, different landing.
- **Don't oversell good results, either.** A good number might be luck. Report it honestly, including uncertainty.
- **Be willing to be wrong in public.** If a stakeholder finds a flaw in your analysis, acknowledge it. "Good catch, let me rerun that" builds more credibility than defensive justification. Goes back to the three phrases — "I'm open to debate on this," "I'm happy to be wrong on that," "but of course, I could be wrong."

**Write the impact statement for every project.** One page. Business question, approach, results, measured impact. This document is how you defend the team's budget next year, how individual contributors build a portfolio for promotion, and how you force the question "did this actually matter" to be answered in writing. If you can't fill out the impact statement, that's telling you something.

**For portfolio and student work:** you still write the impact statement, even with invented numbers. "I'm pretending this classifier is for a bank's fraud team. A false negative costs ~$450 in average fraud loss. That's why I'm weighting recall heavily. If deployed with the pilot parameters I used, it would catch approximately 72% of fraudulent transactions in the test distribution, which on 1M annual transactions with a 0.3% fraud rate translates to about $970K in prevented losses." Fictional, but it builds the muscle. The muscle is what separates someone who's industry-ready from someone who just has a degree.

## A word on notebooks

Notebooks are fine for exploration and teaching. They are not fine as the final artifact of a real project. You can't deploy a notebook. As complexity grows, shift to Python as an object-oriented language, running from an IDE and a CLI, with real class structure, proper logging, and the ability to run on a schedule. The earlier this habit forms, the easier the transition to professional work will be.

For exploratory work within a session, a notebook-style script is fine. For anything the user will run more than once or share, write real modules.

## When the user is a student or new practitioner

A lot of people who will use this skill are students or career-changers. Some context that helps:

- Academic data science is broken in specific ways: data handed to you clean, model pre-selected, one metric to optimize, results graded against a known answer. None of that is how real work goes.
- It's normal to feel unprepared coming out of a program. The gap is the industry's worst-kept secret — hiring managers know it, which is why new grads often come in at intern levels.
- The fix is building a toolbox through self-study on real, messy data. Get your own datasets, do your own EDAs, write code from scratch, struggle through it. That's how tools actually enter your toolbox.
- Python is not data science. A Coursera neural networks course is not data science. Knowing libraries is not data science. Data science is the process of applying the scientific method to data.
- "Maybe I'm wrong" is one of the most powerful things a data scientist can say. Defensiveness is a tell for someone who doesn't understand science yet.

If the user seems to be in this boat, be a best friend about it — honest, kind, constructive. Don't pile on. Meet them where they are and walk them through the process on whatever project they've got.

## Output expectations

When helping with a data science task, your responses should:

- **Name the phase.** "We're in Ideation" or "This is a Data Enrichment question." Makes the process visible.
- **Ask the phase-appropriate questions** before writing code. For Ideation: what's the business question, who's the stakeholder, what's the target metric, how will we measure impact? For Data Sourcing: where does the data live, what's the refresh cadence, who owns it, what's the data dictionary? For EDA: have we looked at distributions, missingness, target balance, and temporal patterns? For Model Design: what's the CMS plan, what models are we trying, what's the holdout strategy?
- **Always do EDA before modeling** on a dataset you haven't seen. Even a brief `df.info()`, `df.describe()`, missingness check, and target distribution plot is better than diving straight into a model.
- **Adapt the methodology to the problem type.** Supervised classification/regression is the default narrative, but the same phases apply to unsupervised, time series, anomaly detection, and other problems. See the "Beyond supervised" section for adjustments.
- **Write code once the phase calls for it** — but annotate what phase the code is executing. A preprocessing script should say "# Data Enrichment — adding time-of-day features" at the top, not just dive in.
- **Flag mindset traps by name** when you catch them. "That would be a Rusty But Trusty move — let's run a CMS instead."
- **Close with a process summary** when presenting results. Not just the number, but the methodology that produced it — and when possible, translate the result into business terms (hours saved, revenue moved, risk reduced). See the Storytelling section.

## Resources in this skill

- `references/phases.md` — Deep dive on each phase of the methodology, with what to ask, what to do, and what to produce. Read this when helping with a specific phase.
- `references/mindset-traps.md` — Expanded discussion of the four mindset traps with examples from real projects. Distinct from statistical biases (which the user should learn from statistics coursework).
- `references/stakeholder-communication.md` — How to talk to stakeholders, the Excel test, visualization rules for technical vs. business audiences, and how to demonstrate impact.
- `references/cms-template.md` — A concrete template for running a champion model search, including how to structure the experiment tracking and what to report.

Read these files on demand as the conversation requires them — don't try to load them all upfront.