1032 Smart Http Plugin Content Management Fix 9a8ecfe6

# Smart HTTP Plugin Content Management Enhancement

**Version Implemented:** 0.229.001

## Issue Summary
The standard Semantic Kernel HttpPlugin was causing "context length exceeded" errors when scraping large websites, as it loads entire web page content without size limits, often exceeding the model's token capacity (200,000 tokens for o1 model).

## Root Cause Analysis
**Original Problem:**
- HTTP plugin loads entire web content as raw HTML strings
- Large websites (like BBC News) return 960k+ characters (~480k tokens)
- No built-in content size limits in semantic kernel HttpPlugin
- Result: Token limit exceeded errors preventing web scraping functionality

**Technical Details:**
- Azure OpenAI o1 model: 200,000 token context limit
- Large website content: 200k+ tokens after HTML processing
- Error: `This model's maximum context length is 200000 tokens. However, your messages resulted in 231425 tokens`

## Solution Implemented
Enhanced web scraping with **Smart HTTP Plugin** featuring intelligent content management:

### 🎯 **Smart Content Size Management:**
- **Character Limit**: 75,000 characters (≈50,000 tokens) default
- **Intelligent Truncation**: Cuts at sentence boundaries when possible
- **Content Type Detection**: Handles HTML, JSON, and plain text differently
- **Size Validation**: Checks content-length headers before downloading

### 🔧 **HTML Processing Intelligence:**
- **Text Extraction**: Converts HTML to clean, readable text
- **Content Prioritization**: Targets main content areas (article, main, content)
- **Noise Removal**: Strips navigation, headers, footers, scripts, styles
- **Structured Output**: Preserves readability with proper formatting

### 📊 **Key Features:**
1. **Prevents Token Overflow**: Guarantees content stays within model limits
2. **Preserves Usefulness**: Intelligent truncation maintains context
3. **Better Error Handling**: Graceful failure with informative messages
4. **Content-Aware Processing**: Different strategies for HTML vs JSON vs text
5. **Fallback Support**: Falls back to standard HttpPlugin if needed

## Technical Implementation

### Files Modified:
1. **`semantic_kernel_plugins/smart_http_plugin.py`** - NEW
   - Smart HTTP plugin with content size management
   - HTML text extraction and cleaning
   - Intelligent truncation algorithms

2. **`semantic_kernel_loader.py`** - UPDATED
   - Modified `load_http_plugin()` to use Smart HTTP Plugin
   - Fallback to standard HttpPlugin if Smart plugin unavailable
   - Enhanced logging for debugging

3. **`config.py`** - UPDATED
   - Version bumped to **0.228.003**

### Dependencies Added:
```bash
pip install aiohttp beautifulsoup4 html2text
```

### Configuration Options:
```python
SmartHttpPlugin(
    max_content_size=75000,    # Max characters (≈50k tokens)
    extract_text_only=True     # Extract text from HTML
)
```

## Testing and Validation

### Functional Test: `test_smart_http_plugin_content_management.py`
✅ **Test Results: 3/3 passed**

1. **Smart HTTP Plugin Test**: ✅ PASSED
   - Plugin imports successfully
   - Handles simple sites (google.com)
   - Manages large sites (bbc.com/news) 
   - Processes JSON APIs correctly

2. **Fallback Plugin Test**: ✅ PASSED
   - Standard HttpPlugin remains available
   - Graceful fallback if Smart plugin fails

3. **Loader Integration Test**: ✅ PASSED
   - Semantic Kernel loads Smart HTTP plugin
   - Plugin functions available: `get_web_content`, `post_web_content`

### Before vs After Comparison:

**BEFORE (Standard HttpPlugin):**
```
Error: This model's maximum context length is 200000 tokens. 
However, your messages resulted in 231425 tokens (230233 in the 
messages, 1192 in the functions).
```

**AFTER (Smart HTTP Plugin):**
```
✅ Content from: https://www.bbc.com/news

[Clean, readable text content - 75,000 chars max]

--- CONTENT TRUNCATED ---
Original size: 156,789 characters
Truncated to: 75,000 characters
Content type: Extracted text content
Tip: For full content, try requesting specific sections or ask for a summary.
```

## User Experience Improvements

### Web Scraping Now Works For:
- ✅ **Large News Sites** (BBC, CNN, Reuters)
- ✅ **Documentation Sites** (Technical docs, wikis)
- ✅ **E-commerce Sites** (Product pages)
- ✅ **Blog Posts** (Medium, WordPress sites)
- ✅ **JSON APIs** (RESTful endpoints)

### Smart Features:
1. **Content Prioritization**: Focuses on main article content
2. **Readability**: Clean text without HTML noise
3. **Context Preservation**: Intelligent truncation maintains meaning
4. **Error Prevention**: Proactive size management prevents token overflow
5. **Informative Feedback**: Clear truncation notices with tips

## Usage Examples

### Agent Prompt Examples:
```
"Get the latest news from BBC"
→ Smart plugin extracts clean text, stays within token limits

"Scrape the content from this technical documentation page"
→ Focuses on main content, removes navigation/ads

"What's on this JSON API endpoint?"
→ Formats JSON nicely, truncates if needed
```

### Troubleshooting:
- **Large Content**: Plugin automatically truncates with notice
- **Slow Sites**: 30-second timeout with clear error message
- **Network Issues**: Graceful error handling
- **Unsupported Content**: Falls back to raw content with size limits

## Performance Impact
- **Memory**: Reduced memory usage due to content size limits
- **Speed**: Faster processing due to smaller content chunks
- **Reliability**: Eliminates token overflow errors
- **Cost**: Lower token usage = reduced API costs

## Future Enhancements
1. **Configurable Limits**: Per-agent content size settings
2. **Content Summarization**: AI-powered content condensation
3. **Selective Scraping**: XPath/CSS selector support
4. **Caching**: Cache processed content to avoid re-scraping
5. **Content Analysis**: Automatic relevance scoring

## Configuration
The Smart HTTP Plugin is enabled by default when `enable_http_plugin: True` in settings. To customize:

```python
# In semantic_kernel_loader.py
smart_plugin = SmartHttpPlugin(
    max_content_size=50000,    # Increase limit for larger content
    extract_text_only=False    # Keep HTML formatting
)
```

**Fixed in version: 0.228.004**

---

**Impact**: This enhancement resolves the fundamental web scraping limitation, enabling reliable content extraction from any website while preventing token overflow errors. The increased 75k character limit (≈50k tokens) provides substantial content capacity while maintaining safety within the 200k token model limit.