Cookie Database Improvements

The Cookie Collector web service has now been brought back online following extensive database re-structuring.

A little while back we made an announcement that we had suspended the collection of new cookie data via our plug-ins.  This was made necessary due to the increase in demand that was over-whelming our systems with new data.

Since the we have been working hard to speed up the rate at which we can process new data, and we are now delighted to report that we have achieved some significant gains.

We are now able to process new cookie data coming in about a thousand times faster than before.  However we have also taken further steps to streamline our systems.

A lot of the data coming in from the Cookie Collector is effectively ‘noise’ – either information we don’t use, or already have.  We are now able to identify this noise much more efficiently, and get rid of it.  Which means we are processing less data in the first place, and at the same time increasing the value of the information we are gathering – because we can put it into use much more quickly.

So what does this all mean?

Search results for domains held in the database are visibly much quicker.

New cookies and new websites are also being added into the database much quicker – so the domain counter should rise more quickly.

Most importantly it means we are collecting more data now, and creating the increased capacity needed as we expand our range of services in the future.

Anyone using the Collector at the moment to spider their site and then look up the results in our search page, may still experience delays before their site appears in our listing – however it is still there, just being processed.

If you go away for a while and have a cup of tea – you should find your site and the cookies we have found will be there.

Next Steps

We are not stopping there however.  The next phase is to develop a more targeted version of the Cookie Collector plug-in, to be released as a Cookie Auditor – more about that later.