Think Before Coding

To content | To menu | To search

Sunday, April 27, 2014

Monoidal Event Sourcing Examples

Last time we tried to imagine how we could change usual event sourcing definition$ to get a monoidal approach.

 

Here are some examples to make it work in real life.

Monoids properties

We'll try to do it properly, and test that the proposed solutions are proper monoids.

 

Let's recap what defines a monoid:

  • closure of operation
  • associativity
  • neutral element

How to test it

the closure of operation is given by the signature of the operation. In F# it should have a signature like 'a -> 'a –> 'a which represents a function that takes to arguments of type 'a and return a result of the same type 'a.

 

For associativity, we'll use the following test function:

1: 
2: 
let isAssociative (*) x y z =
   ((x * y) * z) = (x * (y * z))

It take a function that will be called '' and three values x y z. It will then test the associativity of '' using FsCheck, a F# port of QuickCheck.

 

FsCheck test the given property for a wide range of values. We'll just have to provide the operator, and FsCheck will check it the associativity holds for a lot of values.

 

For the neutral element, will use the following test function:

1: 
2: 
let isNeutralElement (*) neutral x =
    x * neutral = neutral * x

 

Here we'll have to provide the the operator - called '*' inside the function - and the neutral element.

Trivial cases

There are some obvious trivial cases.

 

Let's try to follow the number of occurrences of a specific event in state. The number of time a user did

something.

 

The map function is simply:

1: 
2: 
3: 
let map = function
    | TheUserDidSomething -> 1
    | _ -> 0

 

The combine operation is then really simple:

1: 
let combine acc v = acc + v

and the neutral element is obviously 0.

 

No need to check it with FsCheck, (N, +, 0) is a monoid..

 

Another a bit less obvious is when an event sets a value while others don't.

 

For instance, let's keep track of the last user address when a user moves.

 

For combination, we'll use the 'right' function, which always takes it's rightmost argument:

1: 
let right x y = y

 

The adress should, of course, not change on other events, and for that, we'll use an option:

1: 
2: 
3: 
let map' = function
    | UserMoved (user, newAddress) -> Some newAddress
    | _ -> None

 

The right function can then be tweaked to take the right argument only when it has a value:

1: 
2: 
3: 
4: 
let right' x y =
    match x,y with
    | x, None -> x
    | _, y -> y

right' has a signature of 'a option -> 'a option -> 'a option, so it's closed on operation.

 

It's associative since, whatever x y z, (x right' y right' z) return the last defined term, however composed.

 

None is the neutral element. Added to the left, it keeps what's on the right, added to the right, it keeps what's on the left.

 

We can check it with FsCheck:

1: 
2: 
Check.Quick (isNeutralElement right' None)
Check.Quick (isAssociative right')

A less trivial case

But what about mixing both ? Some events change the value by increments, while some other set a specific value ? Like a stopwatch that increments with a button to reset it.

 

Can we model this kind of thing as a monoid ?

 

We have an increment or a reset value, let's model it as a discriminated union:

1: 
2: 
3: 
type ChangeReset<'T> =
    | Change of 'T
    | Reset of 'T

 

 

A map function, that map events to state change, would be something like:

1: 
2: 
3: 
4: 
let map'' = function
    | TimePassed duration -> Change duration
    | ButtonPushed -> Reset 0.
    | _ -> Change 0.

 

The first two cases are a direct mapping, for other events, we use the Change 0. values that actually use the neutral element of the underlying monoid. Adding 0 will not change the value.

 

We have here an underlying monoid, here, talking about duration, we use numbers, with addition and 0.

 

But imagine we want to add items to a list, but sometime, reset the list to a specific one like the empty list.

 

we can define a high order combine operation like this:

1: 
2: 
3: 
4: 
5: 
let combine' (*) x y =
    match x,y with
    | Change xi, Change yi -> Change (xi * yi)
    | Reset xv, Change yi -> Reset (xv * yi)
    | _ , Reset yv -> Reset yv

 

It combines to changes as a new change using the underlying monoid operation - called '*' here. It combines changes as a change.

 

The second line states, that a value (Reset) combined with a change will apply the change to the value.

 

But the third line says that when a Reset value is on the right of anything, it overrides what's on it's left.

This operation is by it's signature closed on the ChangeReset<'t> type.

 

It's associative, because while combining changes, it has the associativity of the underlying monoid, and when combining Reset values it has the associativity of the right operation.

 

The neutral element is a change with the value of the neutral element of the underlying monoid.

 

We can verify it with FsCheck:

1: 
2: 
Check.Quick (isNeutralElement (combine' (+)) (Change 0))
Check.Quick (isAssociative (combine' (+)))

General structure

I had some remarks that we needed it to be a group and not only a monoid. The right' and combine function clearly don't define a group because elements don't have a inverse/opposite.

 

What would be the opposite of Reset 5 ? It set the value 5 whatever is on its left.

 

The idea is to take the set of state values S.

 

Then take the set of changes from states in S to other states in S that are represented by the events.

 

Make the union of it. SC = S U C. SC will contain more that valid aggregate states. It will also contain things that are not state, like a value that indicate that the state did not change.

 

But we will ensure the following things: the function that convert to changes will return items of C:

1: 
map: Event -> C 

 

The combine operation will be closed on SC, so it can accept States from S and change from C, and it will be closed on SC: combine:

SC -> SC -> SC

 

But it will also ensure that when a state is given on the left, it will also return a state:

combine:  S -> SC -> S

 

The right' operation ensure this. The state need a value at its start, you can add as many None on the right, it'll still have it's value, and any new value will return this value.

 

For the ChangeReset type, the State is represented by the Reset values -that are actual values- it's S, while changes are represented by the Change values that define C.

 

As long as a Reset value is given at the outmost left, the result will be a Reset value that is part of S.

With this, we don't need a group, but we can do with something that is only slightly more restraint than a monoid, only to keep the semantic of state.

But we need more that a single value !

Of course aggregate state is usually composed of more than a single value.

 

Let's start small and consider a pair.

1: 
type State<'a,'b> = 'a * 'b

 

If 'a and 'b are monoids we can easily combine them :

1: 
let combine'' (+) (*) (xa,xb) (ya,yb) = (xa + ya, xb * yb)

where + is the operation on 'a and * the operation on 'b.

 

The neutral element is simply

1: 
let neutral neutrala neutralb = (neutrala, neutralb)

You can easily check that combine is closed on operation, associative, and that neutral is the neutral element.

 

recursively, you can build a triple as a pair of a pair with a single element (('a,'b), 'c), then a quadruple and any tuple.

 

Since a structure - a record - is just a tuple with names, any struct with monoidal members is also a monoid.

 

And since in Event Sourcing, all the tricky decisions are done before emitting an event, applying the event should just be setting values, incrementing / decrementing values, adding / removing items to sets or lists, all those operation are monoidal, hence almost all aggregates should be convertible to monoidal Event Sourcing.

 

Of course it doesn't mean you should do Event Sourcing this way, but it was rather funny to explore this !

 

Have fun !

Friday, April 11, 2014

Monoidal Event Sourcing

I Could not sleep… 3am and this idea…

 

Event sourcing is about fold but there is no monoid around !

 

What’s a monoid ?

 

First, for those that didn’t had a chance to see Cyrille Martaire’s  fabulous explanation with beer glasses, or read the great series of post on F# for fun and profit, here is a quick recap:

 

We need a set.

Let’s take a simple one, positive integers.

 

And an operation, let say + which takes two positive integers.

 

It returns… a positive integer. The operation is said to be closed on the set.

 

Something interesting is that 3 + 8 + 2 = 13 = (3 + 8) + 2 = 3 + (8 + 2).

This is associativity: (x + y) + z = x + (y + z)

 

Le last interesting thing is 0, the neutral element:

x + 0 = 0 + x = x

 

(N,+,0) is a monoid.

 

Let say it again:

(S, *, ø) is a monoid if

  • * is closed on S  (* : S –> S > S)
  • * is associative ( (x * y) * z = x * (y * z) )
  • ø is the neutral element of * ( x * ø = ø * x = x )

warning: it doesn’t need to be commutative so x * y can be different from y * x !

 

Some famous monoids:

(int, +, 0)

(int, *, 1)

(lists, concat, empty list)

(strings, concat, empty string)

 

The link with Event Sourcing

 

Event sourcing is based on an application function apply : State –> Event –> State, which returns the new state based on previous state and an event.

 

Current state is then:

fold apply emptyState events

 

(for those using C# Linq, fold is the same as .Aggregate)

 

Which is great because higher level functions and all…

 

But fold is event more powerful with monoids ! For integers, fold is called sum, and the cool thing is that it’s associative !

 

With a monoid you can fold subsets, then combine them together after (still in the right order). This is what give the full power of map reduce: move code to the data. Combine in place, then combine results. As long as you have monoids, it works.

 

But apply will not be part of a monoid. It’s not closed on a set.

 

To make it closed on a set it should have the following signature:

 

apply: State –> State –> State, so we should maybe rename the function combine.

 

Let’s dream

 

Let’s imagine we have a combine operation closed on State.

 

Now, event sourcing goes from:

 

decide: Command –> State –> Event list

apply: State –> Event –> State

 

to:

decide: Command –> State –> Event list

convert: Event –> State

combine: State –> State –> State

 

the previous apply function is then just:

apply state event = combine state (convert event)

 

and fold distribution gives:

 

fold apply emptyState events = fold combine emptyState (map convert events) 

 

(where map applies the convert fonction to each item of the events list, as does .Select in Linq)

 

The great advantage here is that we map then fold which is another name for reduce !

 

Application of events can be done in parallel by chuncks and then combined !

 

Is it just a dream ?

 

Surely not.

 

Most of the tricky decisions have been taken in the decide function which didn’t change. The apply function usually just set state members to values present in the event, or increment/decrement values, or add items to a list… No business decision is taken in the apply function, and most of the primitive types in state members are already monoids under there usual operations…

 

And a group (tuple, list..) of monoid is also a monoid under a simple composition:

if (N1,*,n1) and (N2,¤,n2) are monoids then N1 * N2 is a monoid with an operator <*> ( (x1,x2) <*> (y1,y2) = (x1*y1, x2¤y2)) and a neutral element (n1,n2)…

 

To view it more easily, the convert fonction converts an event to a Delta, a difference of the State.

 

Those delta can then be aggregated/folded to make a bigger delta.

It can then be applied to a initial value to get the result !

 

 

The idea seams quite interesting and I never read anything about this.. If anyone knows prior study of the subject, I’m interested.

 

Next time we’ll see how to make monoids for some common patterns we can find in the apply function, to use them in the convert function.

Saturday, January 4, 2014

Event Sourcing. Draw it

Here is a drawing to show the interaction between the Decide and Apply functions:

Tuesday, December 17, 2013

Generalized Conway's law

Disclaimer: This article is not just about Conway's lay which I'd call Classical Conway's law, it's already on Wikipedia. You'll find its Generalized form at the en of the post..

BuildStuff 2013 has been really inspiring. Talking with all those people from different CS culture always raise the level of the conversation.

 

Several times came Conway's law into the discussion, as Pieter Hintjens noticed in his blog post yesterday.

 Classical Conway's law

Conway's law is quite simple. It states that:

organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations

 

À partir de l’adresse <http://en.wikipedia.org/wiki/Conway%27s_Law>

 

It's often seen as a joke to mock companies organization, like this famous drawing:

 

 

But it is more than that.

First, it can be a force to drive a change in the system.

 

When two parts are too strongly tied and mangled together, it's a good idea to move teams apart. Communication friction will increase, and eventually, teams will find way to decouple parts so that friction can be reduced.

 

Pieter's example of LSHS vs LSNS is totally an example of this.

 

Second, it is both ways.

A messy system structure reflects a messy team organization.

But we can go far beyond.

 

Generalized Conway's law

System should match the structure of any external concern it interacts with

 

Of course, the Conway's law is included here. There's a match between people building the system and it's structure.

 

But we can here also include design methods like Domain Driven Design which address a different concern the same way:

Align the system structure with the structure of the Domain you're working in/for

 

Using Domain Events and Event Sourcing is a way to make explicit something that is usually implicit in most systems: Time

 

CQRS will reflect the fact that outside the system, getting information and acting on it are two totally different things.

 

Architecture in general is also included: find the match between your system and non functional requirements.

  • Deployment
  • Security
  • Performance
  • Maintainability

At a lower level, it is also known that code clarity reflects the level of the developers understanding.

 

All those external forces will shape the system, and any mismatch in structure will create friction.

And any smell in the system is a sign that something is not understood or working well outside of the system.

 

Generalized Conway's law gives a higher frame of thought to find what's ok and what's wrong in all the dimensions of building systems in interaction with their environment.

Monday, July 29, 2013

Command Events and Context

A remark by @serialseb after my previous post :

 

First, here is some the distinction between a Command and an Event.

A command express what you want to do. Usually expressed with a verb in the present tense:
WithdrawMoney

A event express what actually happened. Usually expressed with a verb in the past tense;
MoneyWithdrawn

As you can anticipate, what you request is not always what actually happens:
WithdrawMoney($100000000)

A Command is still an Event ?

Let’s go a bit meta.

The fact that you decide you want to do something is something that happens in time. So it can also be considered as an event !

True… but what’s the usefulness of this ?

Someone who would like to study the correlation between your desires and their realizations will model both as events..

This can be the same in a tracking context. Track both causes and effects..

Should we just ignore the distinction then ?

Surely not !

It’s here again a matter of context. Of Bounded Context.

Let’s take a sample with different kind of concepts, Value Object and Entities, to highlight how the same thing can take different modeling forms in different contexts:

Money is usually modeled as a Value Object, which implies equality by Value. I give you $10 you give me $10, we’re even, even if I give you a bill and you give me some coins.

In the context of fraud tracking, this can be very different. Bills have a tracking number, and you can model it as entity.

If you make both context work together you’ll go through an Anti Corruption Layer for context mapping. You’ll not model all your accounting with entities for money because some other context models it this way !

In the case of Command and Events, some contexts can be interested in the fact that a decision was made, so this will be modeled as an event. But in the context where this decision happens, the decision is different from the actual outcome, and not being clear about this difference can lead to several issues..

For example if you model a withdrawal with  a Withdrawal concept: It is not clear whether it represe,ts the fact that you asked it to happen or the fact that is has actually been done.

Being explicit on it with WithdrawMoney / MoneyWithdrawn removes the ambiguity.

In the context of event sourcing, this distinction is very important as the previous article showed. As is the distinction between upstream events and internal events, since upstream events is input and will only produce things you’d like to happen, so are closer to commands from the system point of view.

Of course if your context is not about time and decision, don’t use these concepts that would be useless. You’re writing a compiler ? Forget about all this.

Should I have Command DTOs ?

not necessarily, a command can be modeled as a function/method call. So creating Command object is not mandatory.

It can still be useful for dispatch, storage for diagnostics, functional test etc.

Sunday, July 28, 2013

Event Sourcing vs Command Sourcing

Last month, I made a presentation about Event Sourcing - a shorter version of my DevoxxFr talk. After me, Etienne and Maher from Sfeir, did a presentation on the same subject and their architecture inspired by LMAX.

I immediately noticed their reference to this Event Sourcing page by Martin Fowler, and started to see several point of pain lurking in their long term maintenance…

I won’t make a flaming post against Martin Fowler that has written lots of interesting stuff. Even this article says nothing wrong. It just takes a way that can cause long term pains as expressed in the page itself…

Sourcing Events, but which Events ?

The article starts with a totally valid definition of Event Sourcing:

Capture all changes to an application state as a sequence of events.

The question then is… where do these events come from ?

In this case when the service is called, it finds the relevant ship and updates its location. The ship objects record the current known state of the ships.
Introducing Event Sourcing adds a step to this process. Now the service creates an event object to record the change and processes it to update the ship.

A C# sample a bit further to make it clearer:

class EventProcessor...
IList log = new ArrayList();
public void Process(DomainEvent e) {
e.Process();
log.Add(e);
}

As you can notice, the Event is produced before reaching the event processor…

Constrast this with the following version:

class Cargo
{
IList log = new List();
private State currentState;
public Cargo(IEnumberable events)
{
foreach(var @event in events)
Apply((dynamic) @event);
}
public void Arrive(Port port)
{
// logic to verify the action can be done
// based on current state and command parameters
if (IsAlreadyInPort) throw Exception();
// create an event of what happened with this action
// it should not mutate state,
// but it can capture external state when it arrives
// it's also based on current state and command parameters
var @event = new ShipArrived(id, port, DateTime.Now)
// apply change du to the event
// it should require only current state and
Apply(@event);
log.Add(@event);
// events will be published to the rest of the system
// from there.. This is where further side effect will
// occure
}
private void Apply(ShipArrived @event)
{
// no decision should happen here !
currentState.Port = @event.Port;
currentstate.LastMove = @event.Time;
}
}

From a functional point of view this pattern can be build from two pure functions:

Decide:
Command -> State -> Event list
ApplyStateChange:
State -> Event -> State

Here, stored event has been produced by the aggregate itself ! The output is stored.

Nice, but why should I care ?

After all, since Martin says the first version is ok… let’s go !

This would be without noticing several warnings in the rest of the article.

External Systems

from the same page:

One of the tricky elements to Event Sourcing is how to deal with external systems that don't follow this approach (and most don't). You get problems when you are sending modifier messages to external systems and when you are receiving queries from other systems.

Many of the advantages of Event Sourcing stem from the ability to replay events at will, but if these events cause update messages to be sent to external systems, then things will go wrong because those external systems don't know the difference between real processing and replays.

The second version doesn’t suffer this problem…

Because rebuilding the state (like done in the constructor) only use the Apply method (or the ApplyStateChange function in the functional version)..

This Apply method only works with internal state and produces no external side effects..

External Queries

Another problem arising with Martin Fowler’s proposal:

The primary problem with external queries is that the data that they return has an effect on the results on handling an event. If I ask for an exchange rate on December 5th and replay that event on December 20th, I will need the exchange rate on Dec 5 not the later one.

Here again, the second version doesn’t suffer the problem..

The data from the external system will be used to build the event. It can be directly stored in it (like the current time in the sample), but can also be used in a computation. For instance, command contains prices in USD, query an current rate from USD to EUR, compute a price in EUR and put it in the event.
The rate at the time of the computation is baked in the event ! No need to remember the rate value afterward, especially no need to complex external system gateway.

It could still be better for debugging purpose to put the used rate explicitly in the event.

But the second version intrinsically handles this issue gracefully…

External Interactions

Both queries and updates to external systems cause a lot of complication with Event Sourcing. You get the worst of both with interactions that involve both. Such an interaction might be a an external call that both returns a result (a query) but also causes a state change to the external system, such as submitting an order for delivery that return delivery information on that order. 

Problem solved by version 2….

Code Changes

So this discussion has made the assumption that the application processing the events stays the same. Clearly that's not going to be the case. Events handle changes to data, what about changes to code?
[…]
The third case is where the logic itself changes over time, a rule along the lines of "charge $10 before November 18 and $15 afterwords". This kind of stuff needs to actually go into the domain model itself. The domain model should be able to run events at any time with the correct rules for the event processing. You can do this with conditional logic, but this will get messy if you have much temporal logic. The better route is to hook strategy objects into a Temporal Property: something like chargingRules.get(aDate).process(anEvent). Take a look at Agreement Dispatcher for this kind of style.

Wooo… when I read this, it’s a red flag for me ! I never want to deal with this kind of problems !

Especially if they’re expected to happen for sure !

How does it go with the second version ?

Events are produced by the code that contains the logic. Before November 18, the events emitted where based on code that charge $10. After, the code charges $15.

When using the Apply method, it doesn’t have to know how much to charge, it’s already in saved events !

There is no need to keep an history of versions of domain logic - except in your source control !

It can even cop with changes far more complex that the one in this sample. In any case, all data needed to compute current state has been put in the event.

Correcting logic bugs

One of the advantages advanced by Martin Fowler, is that you can change how you take the decision after the fact.

But if an event is an event, it already happened, and there’s no way we can go back in time to change it. We wont be able to change external side effects anyway, so just accept it.

It’s still possible to apply compensations… like accountants. When they charged you to much, they don’t use a time machine to make has if nothing happened.. the just add a chargeback entry at the end of the ledger.

Command Sourcing ?

I call the pattern proposed by Martin Fowler Command Sourcing.

A Command is a request made to the system to do something. At this point a lot of thing can still happen. It can fail, it can be influenced by external state..

An event is something that happen and that cannot be changed.

You can protest that an Arrival Event is an event, not a command.

Sure, but for the system it’s an upstream event. Not something that happened in the system.

Where’s the difference in the second verions ?

The upstream version will go to a reactor that will produce an Arrive command (notice the present tense) inside the system.
The system will then produce a ShipArrived event (notice the passed tense). This event has been emitted by the system itself.

The Martin Fowler’s version takes a shortcut: bypassing the reactor emitting a command, but it is equivalent to sourcing commands.

Martin, this is a call to action !

Since a lot of people have read, and will read the entry on your web site, can you add something about the pattern described here to reduce the pain of people that will want to start with Event Sourcing ?

Thursday, April 4, 2013

C# Static interfaces - Take 3 - redirect

The actual post has been moved to a new address

C# Static interfaces - Take 3

You may believe it or not, but the post that drains most of the traffic of this blog, is the one about C# static interfaces !

 

In october 2009, I simply tried to imagine where the idea of C# static interfaces could lead us, and, since then, I have more viewed pages (> 15%) on this post than on my home page !

 

And since then, nothing moved in this area in the C# langage, and I don’t expect it to happen soon.

 

But some other thing happened…

 

F#

 

Yes F# is out and running on almost all platforms, and it can do what I described in the previous post.

 

The thing is called Statically Resolved Type Parameters and is closer to C++ templates than from C# generics.

 

The trick is that you can define an inline function with statically resolved types, denoted by a ^ prefix. The usage of defined methods on the type is not given here by an interface, but by a constraint on the resolved type :

let inline count (counter: ^T) =
    let value = (^T: (member Count : int) counter) 
    value

here , the count function takes a counter of type ^T (statically resolved).

The second line express that ^T actually should have a member Count of type int, and that it will call it on counter to get the result value !

 

Magic !

 

Now, we can call count on various types that have a Count member property like :

type FakeCounter() =
    member this.Count = 42;

or

type ImmutableCounter(count: int) =
    member this.Count = count;
    member this.Next() = ImmutableCounter(count + 1)

or

type MutableCounter(count: int) =
    let mutable count = 0
    member this.Count = count;
    member this.Next() = count <- count + 1

without needing an interface !

For instance :

let c = count (new FakeCounter())

True, this is compile time duck typing !

 

And it works with methods :

let inline quack (duck: ^T) =
    let value = (^T: (member Quack : int -> string) (duck, 3))  
    value

This will call a Quack method that takes int and returns string with the value 3 on any object passed to it that has a method corresponding to the constraint.

And magically enough, you can do it with static methods :

let inline nextThenstaticCount (counter: ^T) =
    (^T: (member Next : unit -> unit) counter)
    let value = (^T: (static member Count : int) ()) 
    value

this function calls an instance method called Next, then gets the value of a static property called Count and returns the value !

It also works with operators :

let inline mac acc x y = acc + x * y

notice the signature of this function :

acc: ^a -> x: ^c -> y: ^d ->  ^e
    when ( ^a or  ^b) : (static member ( + ) :  ^a *  ^b ->  ^e) and
         ( ^c or  ^d) : (static member ( * ) :  ^c *  ^d ->  ^b)

It accepts any types as long as they provide expected + and * operators.

 

The only thing is that a specific implementation of the function will be compiled for each type on which it’s called. That’s why it called statically resolved.

 

You can use this kind of method from F# code but not from C#.

 

Anyway…

No need for static interfaces in C#, use F# !

Wednesday, April 3, 2013

Time is Money! DevoxxFr

Last week I made a big 3 hour presentation (oops I forgot the pause..) at the DevoxxFr conference.

 

It was the second edition of this big Java oriented event in Paris, and a big success: 1400 attendees, 180 speakers for 160 presentations.

 

I’ve seen a lot of interesting talks there and met nice people ! I can sincerely recommend it to anyone, even not fluent in Java.

 

I posted the slides of my talk on SlideShare – french only.

 

I’ll post a link the the recorded video as soon as it’s available.

 

The slides don’t contain the details of the F# live coding of a Uno game, but it was quite similar to my SimpleCQRS F# implementation on GitHub.

 

Someone noticed it was a F# presentation in a JVM conference while there was no F# at the last TechEd…

Sunday, March 3, 2013

Entities and Repository injection - follow up

 

I have still new comments and there are some Stack Overflow questions on the subject that show the response to the question is still unclear…

 

Should you inject, or even use repositories in you entities ?

The response is two fold, and both go in the same direction, you shouldn’t.

 

Of course, we’re discussing design here, so if in your specific case you find reasons more important than those invoked here, there’s always a possibility to decide to go as you wish…

 

Injection and DDD patterns

Where talking Injection and Entities/Repositories here, so the first reason concerns what injection means in a Domain Driven Design environement.

 

The wikipedia definition of the dependency injection pattern only makes references to objects or components without defining explicitly which kind of object or which kind of component…

 

But a famous injection anti pattern can give use more information on what are injected components.

I call the Service Locator anti pattern.

 

This make it clear that what we are looking for is a way to find service in a supple and configurable way.

I won’t explain here why the service locator is an anti pattern, but it makes it obvious that the use of injection is to provide access to services in the code.

 

Since services are supposed to be stateless, their lifecycles are usually quite flexible. A service could be instanciated on each call, on each request or as a single instance without much difference. The injected service dependencies enable to compose services to replace implementations by configuration or for tests..

But even at runtime. A live configuration change could indicates to replace an injected service on next instantiation.

 

Services and repositories are quite obviously not injected with Entities/Aggregates:

  • A repository is here to instantiate entities, it’s its own job, it doesn’t need to be injected with entities…
  • When a service needs an entity, it can be injected with a repository, and get the entity through it.

But could entities be injected with services or repositories ?

 

An aggregate or an entity is not stateless as a service, it is statefull, and its member fields are here to maintain current state.

It seems quite odd to use those fields to store references to services that are obviously not part of the state.

It also links the referenced service lifecycle to the lifecycle of the entity itself.

 

Most entities instanciation schemes in web applications are on a per call basis, and since most web application don’t do anything between calls, the response to the lifecycle problem is simply that everything in created and destroyed on each call.

But it is only a specific simplistic way to consider object lifecycle. What if you kept your entities longer in memory and reused them on next call to avoid roundtrips with the data storage ?

Having a reference to a service inside an entity state would make the live reconfiguration far harder to achieve : You would have to trash all current state and reload everything. There is no such problem with services since they’re meant to be stateless.

 

Entities fields are meant to store state, no to reference services. Their lifecycles should not be coupled.

 

Consistency boundary

 

The second reason is about the aggregate consistency boundary.

 

Why would you typically need a reference to a repository inside an aggregate ?

 

First reason would be to load sub entities when needed. It’s also called delay load.

 

You shouldn’t need delay load in aggregates

The reason comes from the aggregate definitions. From wikipedia:

Aggregate: A collection of objects that are bound together by a root entity, otherwise known as an aggregate root. The aggregate root guarantees the consistency of changes being made within the aggregate by forbidding external objects from holding references to its members.

 

The definition talks about object being bound together. Those objects – entities – have strong relationships, or grouping them in an aggregate makes little sense. When the relation is strong, you will need for each

aggregate method to change all objects in coordination, or none. If not, parts of your aggregate should not be here.

If an aggregate needs all its state for each operation, there is strictly no reason to use delay load.

Load all, use all, save all.

 

The second reason would be to load entities in other aggregates.

You shouldn’t need references to entities in other aggregates

or at least not directly. The service calling the method on the aggregate will find information needed to call the method (which can contain information coming from other aggregates), but if you need a direct reference to another entity, it’s a clear indication that the aggregated boundary is wrong ! Put that other entity in the aggregate !

 

Thus once your aggregate modeling is ok, there is really no reason to use a repository inside an entity.

Thursday, June 21, 2012

SimpleCQRS the F# version

Here it is ! As promised after greg’s talk at dddx  (you can already see all the presentations online !), a F# version of SimpleCQRS, a simple, quick EventSourcing + CQRS sample to see it in action.

 

Why rewrite it in F# ?

 

This is not just a simple copy of the C# version. The point was first to write it in a functional language, because event sourcing is inherently functional.

 

In C#, an aggregate method looks like this :

 

public void CheckIn(int count)
{
   if(count <= 0)
        throw new InvalidOperationException(
              "must have a count greater than 0 to add to inventory");
   ApplyChange(new ItemsCheckedInToInventory(_id, count));
}

 

The ApplyChange method is defined in the AggregateRoot base class.

It dispatches the event to a state denormalizer and add the Event to uncommitted changes.

 

Here is a denormalizer to apply a state change following an event :

 

private void Apply(InventoryItemDeactivated e)
{
   _activated = false;

 

All this is fine, but why should the aggregate be mutable when the event stream is highly a append only store of immutable events.

 

Watch greg’s talk carefully, both methods can be transformed to an immutable equivalent easily.

 

First the goal of the public method is to determine which event to raise based on command parameters and current state. The CheckIn method can be defined by the following method signature:

int –> State –> Event  or Func<int,State,Event>

 

Instead of calling an Apply change internally, the method simply returns an event :

 

let checkIn count s =
    if count <= 0 then
       raise (InvalidOperationException
           "must have a count greater than 0 to add to inventory")
       fire {ItemsCheckedInToInventory.Id= s.Id; Count = count}

 

where the fire function simply creates a Event array from a single event :

 

let fire o =
    [o :> Event]

(notice the [o:> smiley here !)

An array is returned here so that a method can fire/return several events using a simple :: syntax.

 

The state application can also become immutable. The function gives next state based on event and previous state :

 

State –> Event –> State or Func<State,Event,State>

 

let applyOnInventoryItem s (e: Event) =
   match e with
   | :? InventoryItemCreated as e -> {Id = e.Id; Activated = true }
   | :? InventoryItemDeactivated as e -> {s with Activated = false; }
   | _ –> s

no need for several methods here, every thing is straight forward. Match the event using its type.

The first event is a creation, so a state record is creates.

The second events copies state s with Activated set to false. No change occurs here, a copy is returned.

The _ match specifies that any other event simply return previous state.

 

Done.

 

The current state is a left fold of passed events

Sure and the replayWith method is simply here to do this :

 

let replayWith  =

    Seq.fold

let replayInventoryItem =

    replayWith applyOnInventoryItem { Id = Guid.Empty; Activated = false}

 

the replayInventoryItem is a function that takes a Event seq aka IEnumerable<Event>. It will start with the empty state, then for each event, call the applyOnInventoryItem function with previous state, current event, and iterate with new state.

 

The result is the current state.

 

Command handlers

The event handlers use the following functions :

 

let load id =
   eventStore.GetEventsForAggregate id |>
   replayInventoryItem

let save = eventStore.SaveEvents

let applyOn id version f =
   load id |>
   f |>
   save id version

load simply pass events for the aggregate with identifier id to the replayInventoryItem

save is simply a short cut for the event store SaveEvents method

applyOn loads an aggregate to current state, pass state to f, a function that returns an event seq, then save it to the event store.

 

Here is a sample of its use :

 

member x.Handle (c: CheckInItemsToInventory) =

   checkIn c.Count |>

   applyOn c.InventoryItemId c.OriginalVersion

The checkIn function actually expect an second State argument, it signature is int –> State –> Event seq

 

After passing the c.Count integer argument this is now a State –> Event seq function.

When passed to the applyOn function, the state issued from the load call will be passed to it, resulting in an Event seq that will be passed to the save function.

 

Conclusion

There are a few things to notice in this implementation.

  1. It is very short, much shorter that C# version.
  2. There is no InventoryItem class. The InventoryItem module contains a State record, the representation of the aggregate internal state, functions to determine raised events, and functions to determine next state based on previous state and event. No base class is needed for event dispatch and uncommitted event handling.
  3. There is no repository. Actually the load and save methods do what a repository does, but it’s so simple that no class is required.

I did not talk about the read model that is a bit less interesting here.

Thursday, June 14, 2012

Make it simpler : get rid of Mocking Fx

In the wake of Rinat Technology Demons, I’d add Mocking frameworks to the list.

 

There are several reasons to do that.

 

Stubs

One of the usage of Mocking - or isolation – frameworks is stubbing. A component that access external resources or has complex behavior will be changed for a fake to have simpler predictable results.

 

  • When the component interface is small, the stub declaration code is often more complicated that writing a fake implementation directly.
  • When the interface has many methods, it makes things easier, but here you’ll ignore a code smell. It makes things easy in situation you should not reach !

There are also easy patterns that make the use of mocking irrelevant. A common one is configuration loading.

Lots of developers think: “I need a class to access my configuration. Let’s make a class for this”:

 

    public class Configuration

    {

        public string EndPoint

        {

            get { return ConfigurationManager.AppSettings["EndPoint"]; }

        }

 

        // other properties here//

    }

 

Then they realize they’ll have to isolate from ConfigurationManager, so they add a interface :

 

 

    public interface IConfiguration

    {

        string EndPoint { get; }

    }

 

    public class Configuration : IConfiguration

    {

        public string EndPoint

        {

            get { return ConfigurationManager.AppSettings["EndPoint"]; }

        }

 

        // other properties here//

    }

And they can moke the configuration using a lot of boring code.

 

what about this ?

 

 

    public class Configuration

    {

        public string EndPoint { get; set; }

        // other properties here//

    }

 

    public class ConfigurationLoader

    {

        public Configuration Load()

        {

            return new Configuration

                {

                    EndPoint = ConfigurationManager.AppSettings["EndPoint"]

                    // other properties here//

                };

        }

    }

 

Passing a different configuration in your test is a simple new Configuration {}… and you can use the ConfigurationLoader class in you main function.

 

A good thing here, is that you don’t suffer a read of the configuration file at each call without needing to introduce lazy loading or other tweaks that make things worse..

 

Mocks

When you need to see if a call happened, you can make an implementation of the interface that takes a lambda :

 

    public interface IFancyService

    {

        void DoSomething(int arg);

    }

 

    public class MockFancyService : IFancyService

    {

        private Action<int> doSomething;

 

        public MockFancyService(Action<int> doSomething)

        {

            this.doSomething = doSomething;

        }

 

        public void DoSomething(int arg)

        {

            doSomething(arg);

        }

    }

 

    public class Test

    {

        void DoSomthingHasBenCalled()

        {

            bool wasCalled = false;

            var fancyService = new MockFancyService(i => { wasCalled = true; });

 

            var compoenent = new ComponentThatUseFancyService(fancyService);

            compoenent.MakeSomething();

 

            Assert.That(wasCalled, Is.True);

        }

    }

Or make the fake class record the call, and add a getter to check. It’s not that complicated.

Monday, March 5, 2012

Caching done right

I was trolling on twitter Saturday, when I saw tweet by Nate Kohari and some answers :

 

image

I immediately thought :

If you have one problem and use cache to solve it, you now have two problems.

 

Where’s the problem ?

 

The time to retrieve the data is not negligible due to frequency of request and/or time taken by calculation + data access. So we put data to cache so that we don’t have to endure this time on each call.

 

But then comes the problem of cache expiration:

  • We can use a duration.. but what is the acceptable delay ?
  • We can make a check to original data to check if it has changed. It’s more accurate, but incurs a new data access.

Moreover, checking if it changed is often not enough, we also need to find what changed.

 

And deriving what happened from state is basically reverse engineering. I’m an engineer. I prefer forward engineering.

 

 

Let’s do it forward

 

It’s actually easy, it’s the whole point of CQRS.

 

Let’s build a system that raises Domain Events, and we can denormalize events to a Persistent View Model.

 

We just have to listen to events and change the representation we want to send to the users:

  • The events contain everything we need to do fine grained updates easily.
  • We can can compute denormalizations asynchronously if it’s time consuming
  • We can store it in a relational database, a document database, or in memory
  • We can choose any form of denormalization since it’s a denormalization (object graph, but also flat string, json, html …)
  • It will be up to date quickly because it will be updated when the original data changed
  • The first client that makes a request after a change will not endure a cache miss that can be long to process since computing is done on change, and not on request.

A good way to Keep It Simple, Stupid!

Sunday, February 26, 2012

NuRep your local NuGet+symbols+source repository

As some of you already know, I'm a proponent of DRY : Do Repeat Yourself, but code reuse has some value and nuget is a good way to manage it.

 

So far, the advantages I see in using a package manager are:

  • easy get/update of external projects and their dependencies
  • on demand dependency update (instead of forced dependency update)
  • makes it easier to modularize dependencies

All this thing are out of the box when using OSS projects published on nuget.org, and you can host your own nugets using NuGet.Server.

 

The debugging story is also quite good. symbolsource.org can host symbols and source packages and be used as source server. You can directly step in your favorite OSS source code without having to compile it yourself. They even provide private repositories.

 

But sending your company's source code to an external service is not always compatible with internal policy.

 

In this configuration, using your own nugets leads to a poor dev experience when you have no way to step in your own code : compiled in Release and potentially not your latest code version, indicating the source code in another directory will not give good results.

 

You need a source server.

 

NuRep to the rescue

 

NuRep is a nuget repository based on NuGet.Server but it is also a symbols + code server.

 

When creating your nuget package, specify the –Symbols flag, and nuget will create a .nupkg and a .symbols.nupkg that you can push to NuRep (http://myserver/nurep/api/v2/package).

 

Then configure visual studio's symbols servers (Tools / Options / Debugging / Symbols / add http://myserver/nurep/symbols )

 

Don't forget to enable source server in Debugging / General options, and to disable Just my code.

 

That's it.

 

Now you'll step into the exact code that was used to compile the nuget version.

 

Have fun !

Friday, December 2, 2011

I love SQL Server and cultures... NOT !

When developing a large system, all is not unicorns and rainbows.

For now, everybody was working on a single SQL dev server and friction is high.

 

That’s why I’m working on SQL scripts management with mercurial and powershell to the rescue, so that any developer can trash his own SQLExpress instance, and rebuild everything needed in a single command. (I’ll maybe blog about all that later).

 

We have loads of stored procs.. I know people don’t like it, but it acts as a strong sanity layer when the database schema is so ugly your eyes bleed when you look at it.

 

Yesterday, I run a stored proc, and I get the following error :

The conversion of a varchar data type to a datetime data type resulted in an out-of-range value.

 

Why the f**k.

 

The procedures is using a scalar function :

ALTER FUNCTION [dbo].[DateMaxValue]()
RETURNS datetime
AS
BEGIN
RETURN '9999-12-31 23:59:59.998'
END

 

It’s working on other servers.. why doesn’t it work here.

After several tries, I try with the date ‘9999-12-01’ and I get the following date:

Year: 9999

Month: 01

Day: 12

 

Yes.. the date is interpreted as YYYY-dd-MM on a French server.

 

Even when you use the YYYY-??-?? format, SQL Server still try to mess with culture month/day ordering !

 

You can use the SET DATEFORMAT dmy or SET DATEFORMAT mdy to change this, but it will apply only in current session, and you cannot use it in a stored proc.

 

You can change the server culture, but it wont change anything. The dmy/mdy setting is ultimately in the Login culture.

 

You read it right :

  • For an English Login the function above works.
  • For a French Login the function above fails miserably.

There is no way to my knowledge to specify a strict date parsing in a stored proc or function.

 

So generates your logins with scripts, and enforce the same culture for all logins.

 

It’s just profoundly broken.

Tuesday, June 21, 2011

Event Sourcing and CQRS, Dispatch options 2

In the part one comments, Clement suggested a more efficient solution than registering handler in constructor.

 

The proposed solution is to have a RegisterAllEvents virtual method in which event handler registration would occur. This method is a method instance to have access to this but will be called only once per class. The registration use Expression<Action<T>> to access the expression tree and extract the method info of the handler. This enables type checking, make R# happy – no unused methods – and make reflection not too painful.

 

Good solution.

 

I didn't go that far because with Event Sourcing, you usually keep aggregates in memory, so aggregates are instantiated once per service lifetime.
I just crafted a small performance test :

 

 


using System;
using System.Collections.Generic;
using System.Diagnostics;

namespace AggregatePerfTest
{
class Program
{
static void Main(string[] args)
{
var watch = new Stopwatch();

const int count = 10000000;
Guid id = Guid.NewGuid();

watch.Start();
for (int i = 0; i < count; i++)
new AggregateRegisteredOncePerInstance(id);

watch.Stop();

Console.WriteLine(watch.Elapsed.TotalMilliseconds);

watch.Reset();
watch.Start();

for (int i = 0; i < count; i++)
new AggregateRegisteredOncePerClass(id);

watch.Stop();

Console.WriteLine(watch.Elapsed.TotalMilliseconds);

}
}

public class AggregateRegisteredOncePerClass
{
private readonly Guid id;

private static readonly object ClassInitLock = new object();
private static bool initialized;


public AggregateRegisteredOncePerClass(Guid id)
{
this.id = id;

lock (ClassInitLock)
{
if (!initialized)
{
initialized = true;
// registration happens only once here
}
}
}

public Guid Id
{
get { return id; }
}
}

public class AggregateRegisteredOncePerInstance
{
private readonly Guid id;
private readonly Dictionary<Type, dynamic> handlers =
new Dictionary<Type, dynamic>(5);
public AggregateRegisteredOncePerInstance(Guid id)
{
this.id = id;
Register<int>(OnSomethingHappened);
Register<double>(OnSomethingHappened);
Register<float>(OnSomethingHappened);
Register<long>(OnSomethingHappened);
}

public Guid Id
{
get { return id; }
}

public void DoSomething()
{
Apply(1);
}

private void OnSomethingHappened(int message) { }
private void OnSomethingHappened(double message){ }
private void OnSomethingHappened(float message) { }
private void OnSomethingHappened(long message) { }

protected void Apply<T>(T @event)
{
handlers[typeof (T)](@event);
}

protected void Register<T>(Action<T> handler)
{
handlers.Add(typeof(T), handler);
}
}
}

The code is straight forward, I just created two aggregate classes :

  • one with registration in .ctor based on this post code
  • one without any registration at all, considering that doing it once is the same as not doing it for large numbers, but I added a lock section with a boolean check to simulate what will done on each instance creation.

I created 10.000.000 instances for each, and you get:

  • 3978ms for the one with .ctor registrations,
  • 377 ms for the one without.

It's true that it makes a difference. But how many aggregates do you have in your system ?

 

With 10.000 aggregate you're still under 8ms. I think you can afford that.

 

This is then a trade-off between performance and simplicity :

  • If you have very large numbers, go for expression tree parsing, class lock management etc.
  • In any other situation I recommend using registration in .ctor that makes the code easy to implement in approximately 5min.

Tuesday, June 14, 2011

DDDx 2011

I’m just back from DDDx 2011, and it was great !

The event happened Friday 10 at Skills Matter in London, with great speakers, coffee and food.

 

You can see all the talks on Skills Matter website. Congratulation to the team that released the videos on the web in less that an hour.

 

It was also the occasion to meet IRL DDD practitioners I usually find on twitter.

 

You can also register for next year now for only 50£.

 

So Hurry up !

Thursday, June 9, 2011

Time

556656621_ba9e8c870f[1]

How do we usually manage time in applications ?

Timers, threads, concurrency locks…

If we want to practice Domain Driven Design, we’re surely at the wrong level of abstraction.

 

What is time, btw ?

Tricky question. We know what time is, but… giving a definition is not that easy.

What defines time ? The second ?

Not really. It is used as a measure of time, but it doesn’t seem sufficient.

 

Let’s have a look at Wikipedia’s definition of time  :

Time is a part of the measuring system used to sequence events, to compare the durations of events and the intervals between them, and to quantify rates of change such as the motions of objects. […]

Now we have something interesting : Time is what happens between events.

But what is this thing between events.

The definition of the measure unit surely can give us further insight.

Lest have a look at Wikipedia’s definition of the second :

[…] Since 1967, the second has been defined to be

the duration of 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the caesium-133 atom.

The second is defined as a count of transitions between states of an atom of cesium.

We measure time by considering that the ~time~ between those state transitions is constant.

What if it’s not ?

It’s not that important if other events seem synchronized with those events. Will come back to this later.

 

Events

Let’s step back a bit.

 

How do you feel time passing ?

By looking at your watch ?

 

Maybe, but how could you explain that an hour sometimes seems so long, and sometimes passes in a flash ?

 

Time seems slow and empty when you’re bored.

Times seems fast and full when you’re busy with interesting things.

When you’re bored, it’s because few interesting things happen.

 

You can deduce from this that your personal state transitions are those interesting things that happen.

These are meaningful events. Things that happens and change you deeply.

 

Of course a lot of things happen between those meaningful events, you’re moving, thinking. Your blood flows through your body, but it is just maintenance move. You don’t change deeply.

Maybe some things happen between state transition of a cesium atom, but since we cannot notice it and give it a meaning for now, it has no influence.

 

But when when a meaningful event happens, you change. You’re not the same before and after.

This is what time is about, and this is why it’s one way.

Before –> Event –> After

Events define time by causality

This perception of meaningful events is surely a reason why people say the time pass faster when old. In your 6 first years any event around you is meaningful. Any event make you change since you have no previous knowledge. Then has time goes by, you integrate knowledge and filter things you already know, you’ve already seen. When old, a year can more easily seem the same than the year before.

But some people continue to enjoy and learn as much as they can to still have a long now.

 

When do your system change ?

Your system never change for no reason.

It’s always because a meaningful event happened.

This event can be a user interaction, a call from an external system, a sensor trigger…

And when things change because it’s midnight ?

It simply means that midnight is a meaningful event in your system.

 

Where are those meaningful events in your code ? Hidden in infrastructure code ?

 

I hear Greg Young say :

Make the implicit explicit !

And it’s simple :

Use Domain Events.

 

Once you’ve introduced Domain Events in your domain model, you have made Events and so Time explicit in your domain.

There is no change in the domain that is not due to an Event.

The events appear everywhere in the Ubiquitous Language :

  • When the client HasMoved to a new location, send him a welcome kit.
  • When a RoomHasBeenOverbooked try to relocate the customer
  • Every day at midnight = MidnightOccured, change last minute prices.

I’m sure you can find examples in your own domain. If your domain is business related, it has to deal with time because business is about time and money.

 

Time is now part of your Ubiquitous language and you have an implementation for it.

And that’s huge.

Monday, January 24, 2011

Switching keyboard language in WP7

It was bugging me that I could not switch WP7 keyboard language.

I write most of my emails in French, but my blog and tweets are in English.

I’ve seen that some people were also asking it for the soon to come update.

 

But you can actually already do it easily.

 

Here’s a short how to.

 

Go to Settings/Keyboard. Then tap on Keyboard language.

 

You can select multiple languages here with the checkboxes !

 

That’s all.

 

Then when you open any application with a keyboard you can notice the language selector near the space bar :

imageimage

You an also view all selectable languages with tap&hold :

 

image

 

That’s it.

Friday, January 21, 2011

SmtpListener

I’ve been posting a sample SmtpListener on my repository a few days ago.

It’s infrastructure stuff.. what does it have to do with what I’m usually talking about here ?

 

It’s about Reactive Programming.

 

Sometimes you have to integrate with legacy systems that’ll send you things through email.

Ok, I know, it kind of sucks as an integration mechanism, but still… you have to do this way.

 

The usual way to receive email in an application is to set a mailbox on a mail server (think Exchange), and pull mailboxes periodically to see if there’s something new there.

 

There are two bad things here :

  • People tend to use their enterprise mail server for this. The mail sever is often vital for your business and screwing it up with a bug can have a big impact on your organization.
  • Email are pushed to you.. why would you pull it. You can push it directly on your service bus !

So, I prototyped a small smtp listener that could easily be integrated with whatever you want.

 

You need to add a MX entry in your dns zone configuration so that other servers find it, and get a valid certificate if you want to use secured TLS connections (I’ve disabled certificated verification for demo purpose).

 

But as you can see, the code is very simple since the .Net framework has already what’s needed.

 

The TcpListener is used to receive connections that’ll provide a TcpClient with underlying streams.

The SslStream class is used to encapsulate tcp streams to add Ssl encryption.

I’m using the reactive framework to convert the Begin/EndAcceptTcpClient methods to an Observable to avoid writing the accept loop myself.

 

Then implementation of the protocol is very easy, you can find an overview and sample on wikipedia.

The RFCs can easily be found :

RFC 5321: Simple Mail Transfer Protocol

RFC 3207: SMTP Service Extension for Secure SMTP over Transport Layer Security

 

Of course you can use it freely, and propose changes to make it better since it’s not attack proof.

This code is not resistant to known potential SMTP attacks, including dangling connections, long lines etc..

- page 1 of 4