Hacker NewsStreams to try:
- https://hypermass.io/stream/_J0fm450s - extreme UV images of the sun from SOHO - visual data
- https://hypermass.io/stream/_ATtCoEK - GDACS live disaster data (filtered to just the ongoing items)
The CLI (Apache 2.0, https://github.com/hypermass-io/hypermass-cli) tool allows you to subscribe to multiple live streams of files, with the files arriving in a local folder. The filenames are ULIDs so they can be easily sorted into time order in scripts and code. Publishing works similarly - drop files into a folder and they get read, published and deleted locally. The idea is that by being highly opinionated about the shape of the data (files, delivered in sequence, in real time) a bunch of the logic and edge cases get handled well in one place (things like ordering, retries, hash verification, reconnection, etc) - rather than requiring every subscriber to implement it themselves for every source.
A high-speed version for things like departure boards, train positions is in the pipeline.
Background: About 10 years ago, I worked for a few years supporting and developing some of the UK Rail Open Data systems. As a developer I was frustrated at how fragmented the open data space was, and how every open data system was designed as a blank-page exercise. I realised that a lot of data falls into an odd middle ground, too infrequent to economically poll on a large scale, too large for messaging protocols, but in some sense still real time. For example transport schedules, periodic solar imagery, disaster alerts - real-time but infrequent.
Fairly often I just wanted a "just give me the data" option when the solutions I found were static catalogues with no nod towards real-time use, bespoke APIs, analytics systems and other things I didn't need. I couldn't find a simple approach that leaned into what I saw as the data's strengths (simple files, delivered in order, as/when they are published) - so I built it.
About me: Solo founder based in the UK, full time for just over a year.
Sustainability: Initially a "heavy users pay for more data allowance" model with a decent free tier. Ideally we later shift to institutional publishers pay to keep their streams free to download. Possibly a paid data marketplace to support costs, but the main focus would always be the free and open data.
What I'd like to know:
- Does the CLI approach solve any real problems for you?
- Is the technical approach sane?
- Would you use a service like this to publish data?
- Recommendations to bootstrap users (particularly publishers)?
loading...