The problem is old and small. Bands we like play within driving distance and we find out from a friend, afterwards, with photographs. The streaming service knows exactly who we like: it has the list of people we follow, a ranking of who we actually play, and a clear view of the one thing it has never sold us, which is tickets. So in October the house was given the job of reading the one and looking for the other.
Who counts as "we like"
The streaming service's own token, borrowed read-only from the integration the house already has, gives the top artists over three time spans plus the follow list: about ninety names. We wanted the neighbours too, the bands you would go and see if you knew they existed. The service has an endpoint for exactly that, and it answers hobby applications with a 403, so the neighbours come from a volunteer-run listening database instead, which will tell anyone who asks politely who else the fans of a given band listen to. The identifiers that join the two come from a volunteer encyclopedia of music, one request a second, cached for a month. A hundred and fifty-nine artists in all, checked every other day at twenty past three in the morning.
The encyclopedia turned out to be useful for a second thing. It knows who is dead. A person whose life span has an end date is skipped, which removed a surprising number of hits for guitarists who have been unavailable for decades. A band whose life span has ended is not skipped, because bands come back, and one of ours is doing exactly that next spring. It is only marked suspicious, and anything under a suspicious name with "tribute", "orchestra" or "lightshow" in the title is thrown out, because a band that ended in 1998 and is playing a club in March is almost always four other people with a projector.
Thirty-odd places to ask
Nobody sells all the tickets. The sellers fall into four kinds, in descending order of manners.
- One official API with a key, from the transatlantic giant. It covers a few of our countries and none of the region's largest seller, which is a different company and does not talk to it.
- One semi-official search, the internal endpoint the regional seller's own website uses, in six national flavours. It has no key and answers only a client that looks like a browser down to the language header; a plain library call gets a 403, a hurried series of calls gets a short ban. Its search is soft: a four-letter band name returns 7,313 results, so the house asks for the ranked list and keeps only the exact name. It is paced at two and a half seconds a call, which is why a full run takes most of an hour.
- Two tour widgets that bands embed on their own sites, each with a per-artist identifier. The identifiers are not searchable, by policy, so they are read from the encyclopedia's link list for each artist instead, which is where the bands put them.
- Twenty-odd catalogues: city event guides, venues, promoters, two festivals, a ticket shop in each small country. Each is downloaded once per run and matched locally against our list, because searching a venue's site for a hundred and fifty-nine names is rude and slow, and asking for the whole programme once is neither.
Every catalogue has a canary: the smallest number of events it is allowed to contain. A venue programme with zero events is not a venue with nothing on, it is a page that changed shape, and a scraper that reports "nothing found" when the page has moved is the healthy-looking failure this site keeps warning about. Each source counts its errors too; two runs in a row without a single answer and the house says so, by name, in the morning summary, rather than quietly reporting fewer concerts.
Where "nearby" is
Nearby is a circle, not a country. Four hundred kilometres around home, three hundred and fifty around the second place, a rectangle over a neighbouring country's northern plain, where the big arenas are, a small circle around one capital we pass through often, and a sixty-five-kilometre patch of a flat country up north where we tend to be in December. A city 330 km up the motorway counts; the next one at 520 does not, however good the band. It is the venue's coordinates that decide, which means geocoding, which means a table of the venues we know and a public geocoder at one request a second for the rest.
Then there is travel. The travel calendar, read-only and distrusted, knows which city we will be sleeping in on which nights. A concert in that city on those nights counts even when the city is nowhere near either circle, filtered by taste rather than by our list, because the point of being in a strange city is not to see the same bands. For the trips on the calendar right now the radar found between four and fifteen candidates each, and marked them "on the road" so they do not look like a cheap flight's worth of surprise.
What a date looks like
The region's dates are a rich seam. One seller writes the month in its own language or in English depending on nobody knows what, with the full stop after the day optional; a metal band fell out of the list for a week over that full stop. One arena writes its city in Cyrillic. One site hides the date in a styled span that moves. A band called Death produced a club night called "Death in Rome", which is not Death, and now a match needs the name and a separator after it. A double tribute night to two bands we do like made it all the way to a phone notification before the word "tributes" was promoted from a suggestion to a veto. The first run found twenty-four concerts, of which eleven were not concerts we wanted; the second found fifteen, all real; the catalogues then pushed it to forty, and forty-one is where it sits now, thirteen of them flagged "check" for a human.
The part that is actually useful
New concerts arrive as one notification, listing the artist, the date, the city and which source said so; a concert already seen is not news twice, even when a second seller lists it. Everything else lives on a tab in the house's dashboard: block an artist, mark "we have tickets", and attach the tickets themselves, because the useful question in November is not "is there a concert" but "where did we put the PDF". The house keeps them, sends them to the chat channel the day before and the morning of, reminds a week out, and nudges separately when a concert is in another city and the travel calendar has no trip there yet. The first fruit came within a day: a guitarist from the south, in the second city, in December. Four sets of tickets since, two of them for shows only the giant's API knew about.