Cookie Collector Web Service Update

The Cookie Collector has been busier than ever in the last few weeks, and we now have over 78,000 websites we have collected cookies from since we started gathering data back in May.

It has been a huge success for us, bigger than we had expected, but it has not been without its problems.

The sheer volume of data involved has taken us by surprise, and the cookie database has been re-engineered more than once to handle the amount of data and the rate at which we can process it.

For a while we have been aware that a lot of data being submitted by the Cookie Collector was not getting through because of a processing bottleneck.  As a result of this we developed a two stage system which involved first queuing the raw data and then processing it. 

This solved the problem of lost data, but it then created a situation where the queue was building up faster than we could process it.  In just two days it built up to 7.5 milllion cookies, despite the fact that we were deleting them from the queue as fast as we could process them.

We therefore decided we had to take the service down for a few days, which would give us time to clear the back log. That was last Friday, and it means we have not been collecting any new data since then.

We are working hard on a long term solution but in the meantime the service remains down, which means we are not collecting new cookie data at the moment.  We will of course notify everybody when the situation changes.