Big data issues
IT teams receive regular requests for ad hoc analyses and ‘quick’ reports, but those demanding this information become frustrated because it can take hours or days to get the results they are seeking. And sometimes they don’t get the answers at all.
Those with access to big data sources are demanding the ability to run reports and analyses themselves. Data visualisation is also becoming an increasingly important component of this analysis.
For this to be achievable access to tools that are easy to use and quick to perform the queries are required.
Traveline National Dataset (TNDS)
Basemap has a history of looking at big datasets. We currently produce the Traveline National Dataset (TNDS), which is the digital timetables for all the buses and other minor modes of transport for all of Great Britain. To do this we query the data six times a week and make this available via FTP into OpenData that anyone can consume. It is used by hundreds of organisations and powers transit information in consumer applications such as Google Maps. The raw unprocessed data consists of over 1.6million journeys with a combined total of over 16GB that needs to be processed on each run.
We decided to apply the skills developed in producing the TNDS to a different type of big data; Trafficmaster.
Trafficmaster is free of charge for local authorities in England for use in their highways and planning departments but there are several issues:
- Data was too big to query. This was not a surprise since to look at 5 years’ worth of historical data you needed to load in multiple files and ended up with 100’s GB’s of data.
- Infrastructure costs. The process of querying all the links required expensive hardware and a SQL expert to join and query the data.
- Analysis costs. Overheads of resources and expensive consultants.
This meant that local authorities never used this raw feed of data and relied on the data sent out by the DfT, which didn’t contain all the features, nor was it regularly updated.
Highways Analyst and Trafficmaster
To help Traffic Calming Officers, Network Analysts and Transport Planners within Highways Departments to manage Trafficmaster data Basemap developed Highways Analyst. Highways Analyst is a web tool, which offers a quick, easy and cost effective solution to the big data issues.
It was important to us that Highways Analyst was developed with the users in mind. To gain their input we partnered with South Yorkshire Passenger Transport Executive (SYPTE) and created a forum with 20 representatives from local authorities, public transport authorities and bus companies.
You can find out about the developmental journey of Highways Analyst by downloading Big Data and Basemap
If you would like to see a demonstration of Highways Analyst, please visit our Webinars page for more information.