Rage is a popular all-night Australian music video program broadcast on ABC1 on Friday nights, Saturday mornings and Saturday nights. It was first screened on the weekend of Friday, 17 April 1987.
Since 1998 rage has posted all their episode playlists online. So I created a website that combines these playlists with YouTube allowing you to travel back through time and re-experience rage’s recent history.
Please note: all the matching of track to video is done programmatically so some tracks particularly older ones may not be available (depending on whether someone has uploaded them).
Behind the Scenes (v2 – 2020)
The original version of the site was written using the Lithium PHP Framework in 2012. The site ran well until 2019 when a number of issues cropped up. So I had a plan in mind for a re-envisioned version to address the following issues:
- Over the years Google has been reducing the number of requests that can be made to the Youtube search API. When the site was first developed the project had millions of requests in quota. Now it is down to 100s per month. I tried contacting the black hole that is Google support with no luck. So I needed an alternative to using the Youtube search API.
- The ABC website scraper needs to be rewritten since the ABC site has changed.
- I wanted to make the website static, and have it connect to a serverless backend to avoid having to host and maintain a server.
Given these requirements I decided on the following plan:
- Write a new API backend hosted on Cloudflare workers.
- Dump the playlist data stored in MySQL into flat files (JSON).
- Use scheduled Github workers to scrape the ABC website and update the data directory.
- Convert the frontend to be a static single page application (SPA).
The project is broken into three parts; the API backend, the static frontend SPA, and the ABC scraper script run via Github workers.
For the backend API my initial plan was to write this using Rust (https://www.rust-lang.org/). Cloudflare workers have good support for Rust and I enjoy working with the language.
However I ran into some issues writing a YouTube search scraper and given that there was already a NodeJS package for this purpose it didn’t seem like a great idea to re-invent the wheel. So I ended up writing the final version using Typescript.
The logic is fairly simple when a request is made for a specific music video, it will first check for a cached result stored in Cloudflare KV storage. If the cached data cannot be found it will then go off and grab the data from YouTube.
There is also another endpoint that proxies the data from Github, providing the correct CORS headers for the frontend.
All in all it was fun to re-visit the project.
Checkout the interview with ABC about the site for more details https://web.archive.org/web/20150402201331/https://open.abc.net.au/explore/31540
Source code and data
Scraping the data (particularly the 90s web pages) wasn’t easy due to broken and inconsistent markup so I have posted a database dump in case anyone else wants to play with the data.
Playlists: 1,694
Tracks: 201,316
Approx. Play Time: 1 year, 173 days, 5 hours, 46 mins
https://github.com/patrickgalbraith/rageagain (Source code and data repo)
Changelog
17th January 2020
Features
- RAGEAgain is back after a long hiatus!
- Fixing the site required a complete rewrite. Source code is now available here https://github.com/patrickgalbraith/rageagain.
12th June 2017
Features
- Adjusted Youtube search algorithm to ignore Aria chart information and de-rank reaction and karaoke videos
4th March 2013
Features
- Migrated to new sever (initial load should be significantly faster)
- Finally implemented listing by special
- Updated to jQuery 1.9.1
3rd September 2012
Features
- Updated to include latest playlists
- New special playlist; Top 200 which plays the most played tracks in ascending order (i.e. most played first).
- Individual playlist/tracks are now bookmarkable and browser back button works
- Keyboard shortcuts (spacebar = pause/play, arrow left = prev, arrow right = next)
- Note: If you click on the video you may need to click outside of video for the keyboard events to trigger again. This is because the video will capture any keyboard events.
- Added anchor links to quickly skip to specific year.
- Added warning for users on unsupported/outdated browsers.
- Updated jquery to 1.8.1 (mainly fixes player not appearing in Internet Explorer 8/9)
Bug Fixes
- Fixed issue with tracklist being capped at 200 which caused the first tracks to not be played.
- Massive database overhaul
- Fixed broken 1999 playlists
- Added 30+ missing/incomplete playlists
- Removed a few duplicate playlists
- Removed all duplicate tracks (some tracklists had been entered twice)
- Fixed tracks containing unescaped html entites and tags
29th August 2012
First public release
28th July 2012
Private release
Another one..
Meant to play Taxiride – Everywhere you go
Instead is plays Crowded House.
Bad coding.
Hi Tim,
It looks like an issue with the Top 50 songs because the track name contains the chart position information. I have updated the algorithm to remove this information and adjusted the search ranking.
Hopefully it is working properly again.
– Patrick
Hi,
Is there any way to download the specials and save a copy?
Many years ago when Michael Hutchence died there was a RAGE special with heaps of INXS songs. I taped it on VHS and then years later taped over it. Haven’t heard the end of from the MRS ever since.
I see in your listing by special the INXS special is there so was hoping I could get a copy to end this saga.
Thanks and love the site.
Hi Patrick,
I absolutely love this site but was just wondering how often it gets updated or when you were going to run the scraper again because there’s been some fantastic guest programmers lately. Also, I was wondering why it will briefly play the beginning of the previous song when moving onto the next in the list. I wasn’t sure if it was just on my computer or was something that needed tweaking
Once again,
Cheers for the spanking site
Dan
Hey Patrick,
You haven’t updated Rage Again in a while,
how’s everything going?
Are you going to update this into the future?
Thanks for this service – I use it every weekend. Any chance you will update the site? Seems to have stalled in May.
Hi Patrick thanks for all your work on this, I really enjoy it
Hi Patrick, awesome website. Just wanted to let you know that the top 200 link appears to be broken.
Thanks!
Absolutely love this site but am devastated by it’s inaction lately. If you are too time poor I would be more than happy to take over the reins with a little tutoring so we can get this magnificent site up and running again for alll the multitude of fans out there
Hi Guys,
I’d been keen to open source the site if people are interested in contributing. I will look at writing up some doco and putting it on Github in the next week or so.
Hi Dan,
The site is now back online. I have also made the source code open for contributions here https://github.com/patrickgalbraith/rageagain.
PLEASE get this site up and running again, i know it was working last year and it was the BEST site to rewatch the specials! as abc iview dosent even replay half the music or the specials anymore, just a random dump of 5ish songs from that date… I really hope it gets back up and running, even just for the old episodes.
Hi Tash,
The site is now back online.
You are a fricken LEGEND!!!
I was planning on opening the site up to contributions. However I ran into two major issues.1-Over the years Google has been reducing the number of requests that can be made to the Youtube search API. When the site was first developed I had millions of requests in my account quota. Now it is down to 100s per month. I’ve tried contacting the black hole that is Google support with no luck. So I need an alternative to using the Youtube search API. I have one in mind but it requires a bit of work to integrate.2-The scraper needs to be rewritten since the ABC site has changed.Site is back online after a complete rewrite.
Hi Patrick,
I was wondering if there was something I needed to do with my MacBook pro because every time choose a date or a special it just plays the Rage theme with all the songs scrolling really fast in the top right hand corner with playing a single song, or is the site down again
Thanks so much for this, it’s incredible. Would be great if there was a track navigator, although it might kill the vibe. Cheers boss