Dotnet Techne Serialisation

---
name: dotnet-techne-serialisation
description: Use when selecting serialisation formats and contracts for APIs, messaging, caching, or persistence. Keywords: serialisation, System.Text.Json, source generator, protobuf, messagepack, wire compatibility, AOT.
disable-model-invocation: false
user-invocable: true
metadata:
  author: Metalnib
  version: "1.0.0"
  trigger_keywords:
    - serialisation
    - system.text.json
    - source generator
    - protobuf
    - messagepack
    - wire compatibility
    - aot
---

# Serialisation in .NET

## When to Use This Skill

Use this skill when:
- Choosing a serialisation format for APIs, messaging, or persistence
- Migrating from Newtonsoft.Json to System.Text.Json
- Implementing AOT-compatible serialisation
- Designing wire formats for distributed systems
- Optimizing serialisation performance

---

## Schema-Based vs Reflection-Based (Serialisation)

| Aspect | Schema-Based | Reflection-Based |
|--------|--------------|------------------|
| **Examples** | Protobuf, MessagePack, System.Text.Json (source gen) | Newtonsoft.Json, BinaryFormatter |
| **Type info in payload** | No (external schema) | Yes (type names embedded) |
| **Versioning** | Explicit field numbers/names | Implicit (type structure) |
| **Performance** | Fast (no reflection) | Slower (runtime reflection) |
| **AOT compatible** | Yes | No |
| **Wire compatibility** | Excellent | Poor |

**Recommendation**: Use schema-based serialisation for anything that crosses process boundaries.

---

## Format Recommendations

| Use Case | Recommended Format | Why |
|----------|-------------------|-----|
| **REST APIs** | System.Text.Json (source gen) | Standard, AOT-compatible |
| **gRPC** | Protocol Buffers | Native format, excellent versioning |
| **Actor messaging** | MessagePack or Protobuf | Compact, fast, version-safe |
| **Event sourcing** | Protobuf or MessagePack | Must handle old events forever |
| **Caching** | MessagePack | Compact, fast |
| **Configuration** | JSON (System.Text.Json) | Human-readable |
| **Logging** | JSON (System.Text.Json) | Structured, parseable |

### Formats to Avoid

| Format | Problem |
|--------|---------|
| **BinaryFormatter** | Security vulnerabilities, deprecated, never use |
| **Newtonsoft.Json default** | Type names in payload break on rename |
| **DataContractSerializer** | Complex, poor versioning |
| **XML** | Verbose, slow, complex |

---

## System.Text.Json with Source Generators

For JSON serialisation, use System.Text.Json with source generators for AOT compatibility and performance.

### Setup

```csharp
// Define a JsonSerializerContext with all your types
[JsonSerializable(typeof(Order))]
[JsonSerializable(typeof(OrderItem))]
[JsonSerializable(typeof(Customer))]
[JsonSerializable(typeof(List<Order>))]
[JsonSourceGenerationOptions(
    PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase,
    DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)]
public partial class AppJsonContext : JsonSerializerContext { }
```

### Usage

```csharp
// Serialize with context
var json = JsonSerializer.Serialize(order, AppJsonContext.Default.Order);

// Deserialize with context
var order = JsonSerializer.Deserialize(json, AppJsonContext.Default.Order);

// Configure in ASP.NET Core
builder.Services.ConfigureHttpJsonOptions(options =>
{
    options.SerializerOptions.TypeInfoResolverChain.Insert(0, AppJsonContext.Default);
});
```

### Benefits

- **No reflection at runtime** - All type info generated at compile time
- **AOT compatible** - Works with Native AOT publishing
- **Faster** - No runtime type analysis
- **Trim-safe** - Linker knows exactly what's needed

---

## Protocol Buffers (Protobuf)

Best for: Actor systems, gRPC, event sourcing, any long-lived wire format.

### Setup

```bash
dotnet add package Google.Protobuf
dotnet add package Grpc.Tools
```

### Define Schema

```protobuf
// orders.proto
syntax = "proto3";

message Order {
    string id = 1;
    string customer_id = 2;
    repeated OrderItem items = 3;
    int64 created_at_ticks = 4;

    // Adding new fields is always safe
    string notes = 5;  // Added in v2 - old readers ignore it
}

message OrderItem {
    string product_id = 1;
    int32 quantity = 2;
    int64 price_cents = 3;
}
```

### Versioning Rules

```protobuf
// SAFE: Add new fields with new numbers
message Order {
    string id = 1;
    string customer_id = 2;
    string shipping_address = 5;  // NEW - safe
}

// SAFE: Remove fields (old readers ignore unknown, new readers use default)
// Just stop using the field, keep the number reserved
message Order {
    string id = 1;
    // customer_id removed, but field 2 is reserved
    reserved 2;
}

// UNSAFE: Change field types
message Order {
    int32 id = 1;  // Was: string - BREAKS!
}

// UNSAFE: Reuse field numbers
message Order {
    reserved 2;
    string new_field = 2;  // Reusing 2 - BREAKS!
}
```

---

## MessagePack

Best for: High-performance scenarios, compact payloads, actor messaging.

### Setup

```bash
dotnet add package MessagePack
dotnet add package MessagePack.Annotations
```

### Usage with Contracts

```csharp
[MessagePackObject]
public sealed class Order
{
    [Key(0)]
    public required string Id { get; init; }

    [Key(1)]
    public required string CustomerId { get; init; }

    [Key(2)]
    public required IReadOnlyList<OrderItem> Items { get; init; }

    [Key(3)]
    public required DateTimeOffset CreatedAt { get; init; }

    // New field - old readers skip unknown keys
    [Key(4)]
    public string? Notes { get; init; }
}

// Serialize
var bytes = MessagePackSerializer.Serialize(order);

// Deserialize
var order = MessagePackSerializer.Deserialize<Order>(bytes);
```

### AOT-Compatible Setup

```csharp
// Use source generator for AOT
[MessagePackObject]
public partial class Order { }  // partial enables source gen

// Configure resolver
var options = MessagePackSerializerOptions.Standard
    .WithResolver(CompositeResolver.Create(
        GeneratedResolver.Instance,  // Generated
        StandardResolver.Instance));
```

---

## Migrating from Newtonsoft.Json

### Common Issues

| Newtonsoft | System.Text.Json | Fix |
|------------|------------------|-----|
| `$type` in JSON | Not supported by default | Use discriminators or custom converters |
| `JsonProperty` | `JsonPropertyName` | Different attribute |
| `DefaultValueHandling` | `DefaultIgnoreCondition` | Different API |
| `NullValueHandling` | `DefaultIgnoreCondition` | Different API |
| Private setters | Requires `[JsonInclude]` | Explicit opt-in |
| Polymorphism | `[JsonDerivedType]` (.NET 7+) | Explicit discriminators |

### Migration Pattern

```csharp
// Newtonsoft (reflection-based)
public class Order
{
    [JsonProperty("order_id")]
    public string Id { get; set; }

    [JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
    public string? Notes { get; set; }
}

// System.Text.Json (source-gen compatible)
public sealed record Order(
    [property: JsonPropertyName("order_id")]
    string Id,

    string? Notes  // Null handling via JsonSerializerOptions
);

[JsonSerializable(typeof(Order))]
[JsonSourceGenerationOptions(
    PropertyNamingPolicy = JsonKnownNamingPolicy.SnakeCaseLower,
    DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)]
public partial class OrderJsonContext : JsonSerializerContext { }
```

### Polymorphism with Discriminators

```csharp
// .NET 7+ polymorphism
[JsonDerivedType(typeof(CreditCardPayment), "credit_card")]
[JsonDerivedType(typeof(BankTransferPayment), "bank_transfer")]
public abstract record Payment(decimal Amount);

public sealed record CreditCardPayment(decimal Amount, string Last4) : Payment(Amount);
public sealed record BankTransferPayment(decimal Amount, string AccountNumber) : Payment(Amount);

// Serializes as:
// { "$type": "credit_card", "amount": 100, "last4": "1234" }
```

---

## Wire Compatibility Patterns

### Tolerant Reader

Old code must safely ignore unknown fields:

```csharp
// Protobuf/MessagePack: Automatic - unknown fields skipped
// System.Text.Json: Configure to allow
var options = new JsonSerializerOptions
{
    UnmappedMemberHandling = JsonUnmappedMemberHandling.Skip
};
```

### Introduce Read Before Write

Deploy deserializers before serializers for new formats:

```csharp
// Phase 1: Add deserializer (deployed everywhere)
public Order Deserialize(byte[] data, string manifest) => manifest switch
{
    "Order.V1" => DeserializeV1(data),
    "Order.V2" => DeserializeV2(data),  // NEW - can read V2
    _ => throw new NotSupportedException()
};

// Phase 2: Enable serializer (next release, after V1 deployed everywhere)
public (byte[] data, string manifest) Serialize(Order order) =>
    _useV2Format
        ? (SerializeV2(order), "Order.V2")
        : (SerializeV1(order), "Order.V1");
```

### Never Embed Type Names

```csharp
// BAD: Type name in payload - renaming class breaks wire format
{
    "$type": "MyApp.Order, MyApp.Core",
    "id": "123"
}

// GOOD: Explicit discriminator - refactoring safe
{
    "type": "order",
    "id": "123"
}
```

---

## Performance Comparison

Approximate throughput (higher is better):

| Format | Serialize | Deserialize | Size |
|--------|-----------|-------------|------|
| MessagePack | ★★★★★ | ★★★★★ | ★★★★★ |
| Protobuf | ★★★★★ | ★★★★★ | ★★★★★ |
| System.Text.Json (source gen) | ★★★★☆ | ★★★★☆ | ★★★☆☆ |
| System.Text.Json (reflection) | ★★★☆☆ | ★★★☆☆ | ★★★☆☆ |
| Newtonsoft.Json | ★★☆☆☆ | ★★☆☆☆ | ★★★☆☆ |

For hot paths, prefer MessagePack or Protobuf.

---

## Akka.NET Serialisation

For Akka.NET actor systems, use schema-based serialisation:

```hocon
akka {
  actor {
    serializers {
      messagepack = "Akka.Serialisation.MessagePackSerializer, Akka.Serialisation.MessagePack"
    }
    serialisation-bindings {
      "MyApp.Messages.IMessage, MyApp" = messagepack
    }
  }
}
```

See [Akka.NET Serialisation Docs](https://getakka.net/articles/networking/serialisation.html).

---

## Best Practices

### DO

```csharp
// Use source generators for System.Text.Json
[JsonSerializable(typeof(Order))]
public partial class AppJsonContext : JsonSerializerContext { }

// Use explicit field numbers/keys
[MessagePackObject]
public class Order
{
    [Key(0)] public string Id { get; init; }
}

// Use records for immutable message types
public sealed record OrderCreated(OrderId Id, CustomerId CustomerId);
```

### DON'T

```csharp
// Don't use BinaryFormatter (ever)
var formatter = new BinaryFormatter();  // Security risk!

// Don't embed type names in wire format
settings.TypeNameHandling = TypeNameHandling.All;  // Breaks on rename!

// Don't use reflection serialisation for hot paths
JsonConvert.SerializeObject(order);  // Slow, not AOT-compatible
```

---

## Resources

- **System.Text.Json Source Generation**: https://learn.microsoft.com/en-us/dotnet/standard/serialisation/system-text-json/source-generation
- **Protocol Buffers**: https://protobuf.dev/
- **MessagePack-CSharp**: https://github.com/MessagePack-CSharp/MessagePack-CSharp
- **Akka.NET Serialisation**: https://getakka.net/articles/networking/serialisation.html
- **Wire Compatibility**: https://getakka.net/community/contributing/wire-compatibility.html