Rage is a popular all-night Australian music video program broadcast on ABC1 on Friday nights, Saturday mornings and Saturday nights. It was first screened on the weekend of Friday, 17 April 1987.
Since 1998 rage has posted all their episode playlists online. So I created a website that combines these playlists with YouTube allowing you to travel back through time and re-experience rage’s recent history.
Please note: all the matching of track to video is done programmatically so some tracks particularly older ones may not be available (depending on whether someone has uploaded them).
Behind the Scenes (v2 – 2020)
The original version of the site was written using the Lithium PHP Framework in 2012. The site ran well until 2019 when a number of issues cropped up. So I had a plan in mind for a re-envisioned version to address the following issues:
- Over the years Google has been reducing the number of requests that can be made to the Youtube search API. When the site was first developed the project had millions of requests in quota. Now it is down to 100s per month. I tried contacting the black hole that is Google support with no luck. So I needed an alternative to using the Youtube search API.
- The ABC website scraper needs to be rewritten since the ABC site has changed.
- I wanted to make the website static, and have it connect to a serverless backend to avoid having to host and maintain a server.
Given these requirements I decided on the following plan:
- Write a new API backend hosted on Cloudflare workers.
- Dump the playlist data stored in MySQL into flat files (JSON).
- Use scheduled Github workers to scrape the ABC website and update the data directory.
- Convert the frontend to be a static single page application (SPA).
The project is broken into three parts; the API backend, the static frontend SPA, and the ABC scraper script run via Github workers.
For the backend API my initial plan was to write this using Rust (https://www.rust-lang.org/). Cloudflare workers have good support for Rust and I enjoy working with the language.
However I ran into some issues writing a YouTube search scraper and given that there was already a NodeJS package for this purpose it didn’t seem like a great idea to re-invent the wheel. So I ended up writing the final version using Typescript.
The logic is fairly simple when a request is made for a specific music video, it will first check for a cached result stored in Cloudflare KV storage. If the cached data cannot be found it will then go off and grab the data from YouTube.
There is also another endpoint that proxies the data from Github, providing the correct CORS headers for the frontend.
All in all it was fun to re-visit the project.
Checkout the interview with ABC about the site for more details https://web.archive.org/web/20150402201331/https://open.abc.net.au/explore/31540
Source code and data
Scraping the data (particularly the 90s web pages) wasn’t easy due to broken and inconsistent markup so I have posted a database dump in case anyone else wants to play with the data.
Playlists: 1,694
Tracks: 201,316
Approx. Play Time: 1 year, 173 days, 5 hours, 46 mins
https://github.com/patrickgalbraith/rageagain (Source code and data repo)
Changelog
17th January 2020
Features
- RAGEAgain is back after a long hiatus!
- Fixing the site required a complete rewrite. Source code is now available here https://github.com/patrickgalbraith/rageagain.
12th June 2017
Features
- Adjusted Youtube search algorithm to ignore Aria chart information and de-rank reaction and karaoke videos
4th March 2013
Features
- Migrated to new sever (initial load should be significantly faster)
- Finally implemented listing by special
- Updated to jQuery 1.9.1
3rd September 2012
Features
- Updated to include latest playlists
- New special playlist; Top 200 which plays the most played tracks in ascending order (i.e. most played first).
- Individual playlist/tracks are now bookmarkable and browser back button works
- Keyboard shortcuts (spacebar = pause/play, arrow left = prev, arrow right = next)
- Note: If you click on the video you may need to click outside of video for the keyboard events to trigger again. This is because the video will capture any keyboard events.
- Added anchor links to quickly skip to specific year.
- Added warning for users on unsupported/outdated browsers.
- Updated jquery to 1.8.1 (mainly fixes player not appearing in Internet Explorer 8/9)
Bug Fixes
- Fixed issue with tracklist being capped at 200 which caused the first tracks to not be played.
- Massive database overhaul
- Fixed broken 1999 playlists
- Added 30+ missing/incomplete playlists
- Removed a few duplicate playlists
- Removed all duplicate tracks (some tracklists had been entered twice)
- Fixed tracks containing unescaped html entites and tags
29th August 2012
First public release
28th July 2012
Private release
Have you considered opening this up to user contribution? For instance, having a list of unavailable videos that users could perhaps go back through their Rage VHS tapes and upload to Youtube, then send through a link? A fullscreen option would also be nice which would be possible through HTML5.
Yeah I have thought about this but with 269,938 tracks I’m not sure the best way to go about it.
This is awesome. I love rage but I find it so hard to watch these days, and my spare time rarely coincides with the hours that rage is on.
Will you be updating http://www.rageagain.com with the latest playlists as the weeks go by? Where is the donate button?
Yeah to update all I need to do is run the scraper.
But I need to fix a few small formatting issues with the current database data first.
After that I’ll probably make it run as a weekly cron job which means it will always be up to date.
I have a donate button on this page for a plugin I made http://www.pjgalbraith.com/2011/08/recent-posts-plus/
This is incredible. Top effort mate.
Next step is to make a Spotify App.
They’ve got the API all ready for you and everything!
https://developer.spotify.com/technologies/apps/#resources
Seriously amazing! This would have taken forever, im telling everyone.
Keep up the good work
Fantastic! I’ve been showing/sending Rageagain.com to everyone I know, and everyone is gob smacked how awesome this is.
Simple idea, delightfully executed.
Excellent work dude
a bit jealous that I didn’t think of this idea first
You sir, are a genius!
Sir, you win the Internet. Thank you!
A great tool. How is the video for each title selected? Do you pick the first (most relevant) search result, or are there extra heuristics?
For the past three and a bit years I’ve been manually compiling YouTube playlists of just the new videos played each week. This allows one to listen to every new song played on Rage in about 2 and a half hours a week rather than eleven. I usually make each week’s playlist public at about 4pm Friday, before the Rage broadcast.
You can find the playlists here: http://www.youtube.com/user/NewMusicWeekly
Hi NMW,
The video is selected using the YouTube Data Api (so basically a slightly more complex search query). I have done some filtering to remove irrelevant results however.
Nice work on those playlists. The site I made is kind of the opposite to what you are doing in some ways since I was originally more interested in viewing the playlists from the late 1990s and early 2000s but figured I should make them all work while I was at it.
Patrick
Amazing work, so many memories in these lists. Thanks for bringing them back.
Just a thought, seeing as you have the lists readily available, and the know how. Have you thought about making this into playlists/apps in Spotify. Similar to what Triple J already have for their hottest 100’s?
nice retheme of http://rage.d14n.org/
need to list months in ascending order, add index /search by guest programmer
Hi Daniel,
That’s the first time I’ve seen that site.
Yeah I plan to add an index for specials/guest programmers which should make it easier to skim over.
Hey man,
This is BRILLIANT. A few things, though.
1. I picked a Friday playlist and tried to follow along with the link on the official rage site, except the playlist cut out roughly the first four hours. Any reason why it would be doing that?
2. How can we fix the YouTube link directly going to a live video or even the wrong song? Any way for fans to submit the proper links – not to immediately work or anything, but just for consideration? It’s only happened a couple of times so far, but I’d love for this to be as smooth as possible.
Love it so far, can’t wait for it to get better and better.
I was sooooo excited when a friend sent me the link to this but I can’t get this to work. All I get is the rage intro, again and again. How do I bypass this and get to see the songs? Is there something I’m doing wrong?
Hi Morgan,
Have you tried clicking on the episodes tab in the top left corner?
The episode listing should come up automatically though. What browser/version are you using?
Regards,
Patrick
Good stuff, long time user of rage.d14n.org and ragetube.net here so nice to see a smoother implementation of the idea. Love, love, love the track title overlay.
It would be nice if there was a title somewhere letting you know which show you are watching and perhaps also arrows on either side so you can move backwards/forwards in shows.
One thing I’ve noticed is that that some of the playlists start a good hour or two into the playlist and I know from playing the playlists on the other two sites I’ve linked above that there is youtube videos for those songs. I’ve only played with playlists from 1998 and 1999 so far.
Cheers
Hi Helen,
I did an update today which should fix some of the issues you have been having with the playlist starting late.
I’ve also added keyboard controls which you can use to skip tracks along with the normal menu buttons.
Regards,
Patrick
Patrick, based in my experience manually matcing YouTube videos to Rage playlist items, I’d say the easiest good heuristic for finding the best video for a playlist item would be a naive Bayes technique that assigned points to each factor, then picking the video with the highest total points.
Some factors I’d suggest would be:
1. Points based on the order of search results for a standard relevance search, with the band name and the song name separately quoted. If “featuring/feat.” is in the title, or in case Rage has mispelled, also run the same without quotes.
2. A point advantage where the band name or the record company name is in the video uploader’s name, with a smaller advantage for such videos from VEVO (they have a waternark, and it’s better for a band’s idependence to strengthen their own uploads).
3. A point advantage for more views.
4. A point advantage for higher resolution.
5. A point penalty the longer the video upload date is before the playlist date (earlier uploads are often audio-only).
6. A point advantage for “official” in the title, but a point disadvantage for videos with “making of”, “lyric”, or “live” in the title (unless the Rage playlist item has “live” in its title).
Plus before picking a video you should check that views from Australia aren’t locked-out.
Hi NMW,
Thanks for the great suggestions.
I have sent you an email about this.
Patrick
dude you need to use the full RAGE theme, not the shitty 10 second version, other than that, it’s amazing http://www.youtube.com/watch?v=-tlSsWbF2ww
Wow Thanks for a wonderful contribution, as said you do indeed win at the internet.
Though about a search for artists, resulting in playlists they appear, and with what songs?
A direct link to different playlists straight from the site, so people can choose what might be of intertest?
Also, hi to lifehacker/gizmodo fans!
Just wanted to say.. Patrick, you are a god.
Patrick please call me on *number removed* – would love to chat to you on ABC radio about this.
Elly
This is such a cool idea! I LOVE the design too!! So clean and functional! Awesome work! I remember getting up at 6am on Saturday mornings for rage!
I love this would be even better to have access to the regular youtube control at the bottom of each video to be able to skip to a later section of the videos
You may want to use the Eurasia Bold font for titles, as that is what rage has used since 1997.
A great idea here. If only the ABC would do similar, linked to the actual rage archives instead of the varied quality of Youtube. Of course they would need geoblocking so as to satisfy copyright issues.
Hi CT,
Thanks for the tip about Eurasia Bold.
Yeah it would be great if ABC did that but I have my doubts that they will.
I wouldn’t be surprised if it was just plain impossible due to licensing restrictions even with geoblocking.
On a larger note it seems a shame to have all that content that ABC has produced over the years completely locked away. With the internet it should be possible to stream not just rage but 1000s of episodes of content that ABC has produced for various shows. But unfortunately I can’t see it ever happening…
Patrick
thanks for this site, brilliant idea.
was wondering, the auto link-to-the-best-video suggestion from earlier sounds good, but wouldn’t crowd sourcing be better/faster/easier? eg, devo came up in a certain playlist, the youtube video was an mp3 rip of the song + photo stills rather than the real video. i found a correct version of the video very quickly, would be nice if there was a little ‘I’ve found a better video!” link somewhere.
I imagine it wouldn’t be a huge amount of overhead on your part to approve those and update links. Or if it IS, I bet you’d have a willing army of rage superfans to volunteer to help curate the database.
Hi Matt,
The latest version of the site should improve the video matching (basically I used all of NMW’s suggestions above).
I planned to have a feedback system in-place for this update but I haven’t got it ready yet.
This is really cool. Such an awesome idea
Great idea but I cannot get it to work on either of my two computers. Same problem as “Morgan” above.
I can see the “esisodes” and in the past clicking on one of those played the rage intro only.
Now a few weeks later it doesn’t even do that, the server serves nothing at all (Win 7, IE9)
The “now playing” screen displays a small upper frame and plays the intro only, rest of screen (7/8) is black?
As I click on the different episodes the url in the browsers address bar correctly updates, but the window stays on the episode page and nothing happens at all.
If I then take an episode url and paste it into a reloaded browser it goes to the “now playing” page, plays the 10 second intro at the top of the screen , then nothing.
I don’t have any other issues playing videos.
Any thoughts appreciated.
Hi John,
I have been having issues as well lately with IE9 and the Youtube API. Using the current version of the website along with updating flash to the latest version has improved it for me but occasionally videos will fail to play. At the moment I haven’t had time to investigate the issue in depth but I believe it may be caused by the Youtube Iframe API. So far this issue seems to be only occurring in Internet Explorer.
Patrick
Hi Patrick,
Took your advice and installed Chrome, all is good, works fine.
Perhaps uSoft’s next version of IE will work too, but Chrome is fine.
Thanks again.
I think your parsing of countdown tracks (eg in episode, 17 January 1998) is including its ranking in the chart when searching for its video on youtube.
Cool idea tho
Unfortunately this isn’t working on the Raspberry Pi. I just keeps saying “working” and never gets any further. Would love to see this working on RaspBMC…
Are you still updating the database ? Seems a few weeks behind. Weeks, I tell you !
Oops forgot to implement the scheduled updates after the new year ticked over.
I’ve updated it manually now and it should auto-update every Tuesday from now on.
Cheers, thanks
Just want to say a huge thanks for providing this site. Absolutely awesome and an answer to any unprepared, after the pub, fallback impromptu party tunes.
Cheers!
Great site – just found it. Was hoping there was a way to run on my android tablet which is hooked up to the hifi. However rendering is a little unstable on the android 4.0.3 tablet – is there a way to improve that I am not a techo or anything.
Kudos to you in any case
Will you be providing a more recent database dump?
Hi Malcolm,
Better late than never
Here you go http://www.pjgalbraith.com/wp-content/uploads/rageagain-01-01-2013.sql.zip
Hi – love the idea of this! I’m having the same problem as Stephen (Nov 9 2012 above) was having, in that the ARIA countdown tracks don’t seem to be working, and the obvious answer would seem to be that it’s including the countdown ranking in its search. I’ve noticed this on the Friday 24th December and Saturday 25th December playlists in 1999.
Your plugin is simple and awesome. Thank you for the hardwork, that’s fantastic.
I love Rage Again, but I fear it’s broken. I can’t get it to do anything but play the rage intro these days.
Hi Simon,
It should be fixed now.
Just caused by a hack I added to fix an issue with the Youtube API that stopped working once they finally fixed the issue.
Hurray! It does indeed work again now.
Thanks
Any news when we might get an episode update? Been missing my rage fix whilst away from Australia.
Thankyou Patrick. This is an amazing resource.
Will you be updating it again anytime soon?
Thanks again for all your hard work – I for one really appreciate it.
Great site mate.
but…
It seems that if you are behind a proxy, you cannot play any of the videos.
Any ideas?
Now the website doesn’t work at all!
Proxy or no Proxy.
Any progress on getting the site up and running again? Seems to just skip through all the tracks without playing them. Doh! It was really cool when it was working!
Hi Toby,
Sorry for the long wait just letting you know the site is back online.
Having the same problem as Toby. Would be really cool to get back up and running again.
Hi Evan,
Sorry for the long wait just letting you know the site is back online.
Not working for me either, would be awesome to get it up and running again. Maybe release the core code to github or something public if youve not got time anymore.
Hi Glenno,
Sorry for the wait just letting you know the site is back online. I might consider releasing the source at some point but I would like to rework some things first so I’m not sure when I’ll get around to it.
Hi Patrick,
noticed Rageagain.com isnt working oni fireox, chrome or internet explorer, any ideas on how to fix it?
Thanks
Hi Jarrod,
Unfortunately YouTube changed their API and I need to rewrite the server side code to make it work with the new API. I haven’t had much spare time lately but I’ve started working on it so expect an update in a week or so.
-Patrick
Hi Jarrod,
Sorry for the long wait just letting you know the site is back online.
Mate, Absolute legend! Thanks heaps for offering this!
Hi, This video keeps appearing where our Big league video should be?
Troye Sivan – HEAVEN ft. Betty Who
http://rageagain.com/#/episode/2456/36
Should link to
https://youtu.be/aPIch1w1NzI?list=PL19QdPDM5NK0y6aBkn5Sllf8mOVicNKHt
Thanks from Dave
drummer from Big League
You need to open this up to user recommendations and corrections.
I would say that 50% of each playlist has incorrect songs assigned. Sometimes it’s as innocent as some live version that is play right through to completely incorrect songs… this is especially the case with ARIA Top 50 songs.
Open it up to user correction so we can enjoy this product as it was intended.
As an example to my last post…
The playlist just attempted to play Garbage – When I grow up
Instead it played Pussycat Dolls – When I grow up
Pretty big difference here.