Monday, November 3, 2014

Color of Music II

I long time ago (ages it seems), I posted about mapping the audio spectrum to the visual spectrum after watching Neil Harbisson's TED talk. Well NPR mentioned that same talk today and it got me excited about the project again. I played around some more using my slightly more developed MATLAB and DSP skills, and the results are below.

Quick recap:

Sound and light are both waves though they differ significantly in frequency, medium, propagation type, and speed. A major similarity though is that we have sensory organs for both, and those sensory organs have some interesting quirks. Humans can only perceive small bands of each spectrum. The audio band is typically quoted at 20Hz to 20KHz (point of reference: the A below middle C on a piano is 440Hz). The visible light spectrum goes from roughly 700 nm where infrared transitions to red, to about 400 nm where violet transitions into ultraviolet (it's easier to talk about light in wavelengths=speed of light/frequency because the frequencies are huge numbers). The exact endpoints of these bands vary from person to person, and often with age; however, the range for an average human is well known.

Another quirk of human perception is that we don't perceive equally across these bands. I know a lot more about these limitations in the audio world than in the video world, but there is plenty to read on that if you're interested. We perceive sound on a logarithmic scale, both in loudness and pitch. The decibel scale adjusts for this, so that a 20dB noise sounds twice as loud as a 10dB sound, even though the former is actually about 2.8 times the pressure. And a 40dB sound again sounds twice as loud as that, but is actually 10 times the pressure of 20dB and 28 times the pressure of 10dB. Not only this, but we don't perceive loudness evenly over all frequencies. 

Lower frequencies sound much quieter than middle and high frequencies, and the function is not an elegant relation. Psychologists studied people's perceived loudness and created the equal-loudness contours. Along the red lines are sounds at a constant "perceived loudness" (measured in phons, a made-up unit which sets it's point of reference so that 20dB of pressure equals 20 phons). For different frequencies, different amounts of pressure are required to reach that phon level. There are a number of functions which approximate the correction factor, such as the A-weighting filter.

Pitch perception is logarithmic also. Similar to the ELCs, psychologists created a unit for perceived pitch, the mel (where 1000 mel = 1KHz). A pitch that sounds about twice as high (2000 mel) is about 3.5KHz. The equation is below.


The goal then is to have something that hears a pitch, corrects for proper human perception of that pitch, and then maps that frequency's relative position in the audio band to a color in the same relative position in the visual band. Low frequency sounds would map to red and high frequencies to violet. I want the colors to map to the red-green-blue color space, so that they can be represented by a computer (e.g. an LCD display, a 3-color LED, etc.). Of course, there are all sorts of issues with a computer's ability to generate actual colors, which I touched on in my earlier post about this. However, there are rough approximations that can map the color spectrum to RGB. Here's the one I used from StackOverflow (this guy has awesome graphs and seems to have done some impressive work):

The basic algorithm is to take a FFT of an audio sample at regular intervals to find the amplitude of different frequency components. Then those frequencies are converted to mels for the perceived log scale and then mels are mapped to light wavelengths. These wavelengths are then converted into RGB vectors using the algorithm above. The brightness of each channel is then weighted by both the phasor magnitude of the signal and the A-weighting loudness filter. This results in a final RGB vector for each small sample of audio where brightness is relative loudness and color is relative perceived pitch.

I ran this against MATLAB's sample file (a clip of Handel's Messiah). These are with an FFT sampling rate of 100Hz.

I then generated a video to go along with the audio. The first is the standard result, but it flickered unpleasantly, so the second is with a 10 frame (0.1 second) averaging filter applied. Forgive the low quality of each, generating MPEG4 video in MATLAB is not an easy task, and I ended up stringing together a bunch of JPEGs).



Unfortunately, it seems music is too low-bandwidth to see significant changes in pitch. One thing I might do in the future is map notes to colors, rather than the entire band.

Thoughts? Feel free to leave a comment. :)

Friday, September 12, 2014

New blog on the way

It's been a while since I've posted, but I assure you this isn't blogrot (a term I just made up). I'm in the process of building a new blog, and in the meantime I've been queueing up posts for it. By posting this publicly, I'm basically committing myself to releasing it soon.

I'd like to move all my old posts to the new blog, but that may not be possible, so this might hang around as an archive. In any case, stay tuned.

Sunday, June 8, 2014

Wearable: lighty.. shirty.. thingy

I have a couple of project updates I haven't gotten around to posting, but I wanted to share the one from this weekend first. The neighborhood I grew up in had a midnight 5K which encouraged lights and costumes. I figured this would be a good chance to do some wearable stuff. I spend waaayy to much money at Adafruit and got some EL wire and a decent-sized Neo-Pixel ring. I figured I'd attach these to a hoodie and make some cool light effects (and hopefully win the costume prize! :P ).

I got the aqua EL wire and the cheapest hoodie I could find on Amazon. When I realized the power requirements of the Neo-Pixel and a couple of batteries wasn't going to cut it, I got a decent sized portable charger (10000mAh should power everything for around 1~2 hours) and threw in another strand of EL wire for good measure. This was a different brand and turned out to be much poorer quality. For one thing, it was a log bluer than I expected for aqua. This turned out to be okay, because it's color ended up working better with everything over all, but the two were a lot further off than I expected. The isolation itself was blue; I suspect the wire lights up white underneath instead of lighting the actual color. Also the wire was not as bright and had a "thinner" appearance.

I put off actually putting anything together until the day of the race, so there were some serious hacks. I sewed the Neo-Pixel down to the chest, but broke two needles in the process of threading conductive wire through three through-holes. Then those needed a path to the pouch in the front where the electronics would go. I made the poor choice to "show off" the wires by threading the conductive thread on the outside of the shirt. I did a crappy job in a rush to get it done and it ended up looking pretty bad. One more plug for Adafruit: I had some conductive thread from eBay and it was crap. It was thick and would come unraveled and was generally difficult to work with. That's how I thought all conductive thread was. But I got some more from Adafruit and it was superb: thin and stiff, so easy to thread and tie. I'm not sure how well it would do in a sewing machine, but by hand it was great.

I had planned on attaching this plastic thing (that was actually half of an Easter egg turned upside-down) to act as a partial defuser and weird shape thing to the front, but I had no idea how to attach it. It turned out that hot glue worked much better than expected. Of course, I rushed that too and got some glue where I didn't want it.

As for the EL wire, since the strand from Amazon happened to fit through some holes in the plastic egg thing, I decided to put those on the body, and the Adafruit strand around the hood. This meant that I had to split and solder the cheap wire instead, which was much thinner and an absolute nightmare. The outer wires were a single, crazy-thin strand hidden behind two thick layers of isolation, and I was using a box cutter since I don't own any wire strippers.

I got that down, and realized I needed some sort of bus for the EL wire from the pouch as well. I had some extra EL wire clips so I attached them to some speaker wire (it was easy to tell sides apart and in reach from where I was sitting, so give me a break) and ran that through the inside of the shirt.

I attached the ends of the conductive thread from the LEDs to headers with wire leads. It turns out if you cut away a little insulation and tie the thread around that, it works pretty well. I doused those in too much hot glue and continued on.

At some point, I must have blown out the inverter from Adafruit, because it stopped powering anything. Fortunately the wire from Amazon came with one too. That circuit turned out to be quite resilient. Even though it was designed for 3V, it ran at 5V without anything except an obnoxious high-frequency whine. I after a couple shorts and a couple of shocks (EL wire runs at high voltage AC, if you didn't know), the inverter was happily powering all of the EL wire.

The last physical part was the microprocessor. I used a Teensy 2 since it is my favorite board and I had loaned out all my other boards. At this point it was crunch time, so instead of removing the temporary headers on the Teensy, I just added some more upside down so the female headers from the NeoPixel could plug in. I broke the battery clip on the inverter, so I ended up wrapping the ground from the USB cable to a leg of a BJT. Then I carefully balanced a safety capacitor across it for the LEDs, one leg of which I soldered to a slit in the isolation of the output wire. The battery pack has two outputs, one that maxes around 2A and one that maxes around 1A. I was worried that the LEDs and EL would together pull more than the max of the higher amp output, but it turned out to be okay. I ran the Teensy off the second output (in an attempt to protect it from the sketchy high-power circuit of the other stuff).

With all this working and the rest of the EL wire sewed down, I had about an hour left to program the Neo-Pixel. Fortunately I had played with the library a little bit before. I wanted to make a spinner with a tail of decreasing brightness. The library changes brightness for the entire output, and it is slow and requires a refresh. I tried to write my own function to reduce brightness, but due to some issue with bit shifting and data types that I didn't have time to debug, only the blue worked correctly (since those are the LSBs of the color integer). Blue it is, then. Also, I learned this late that the first LED on the chain doesn't work right. It flashes intermittently and often shows the wrong color. It could be a data/noise issue, but since the rest work, I suspect it is actually just a bad LED. The way it acts is strange, but consistent. I found removing one unrelated test line caused it to intermittently blink blue instead of green. As we say at hackathons: "Fuck it, ship it." I scooped everything up, dug around my closet for running shoes and made it to the race just in time.

One thing I really wish I had more time for was a better way to secure everything. I spent the whole race cradling a few pounds of electronics against my chest to keep them from bouncing around. The LEDs lost data connection early on and froze with a single, off-center white LED on and remained that way the rest of the race. At every bounce, the mode change button on the inverter would get pressed, so the EL wire was blinking randomly, and occasionally the inverter would short and even shock me. I dropped a couple of things and had to go back for them, and that sweatshirt was SO HOT (this is June in Atlanta). I managed to to okay, coming in around 28:02 (and I think I deserve a handicap!).

I didn't win the costume contest, although the judge came by after and said I was his second choice. I managed to put everything back together after the race. I will put the code on GitHub once I've debugged the dimmer issue. A pretty decent way to spend a weekend in my opinion.

By the way, I stumbled across this awesome alternative NeoPixel library for the Teensy 3.0. They made a full display that streams YouTube! Check it. 

Tuesday, February 18, 2014

Swank - A Stupid-simple static webserver

Today doing some web development I got tired of messing with httpd.conf files with XAMPP every time I changed projects, so I whipped up a little project called swank. Simply run swank from a terminal, optionally giving it the root path (if not the current working directory) and the port (if not 8000). It will spin up a static file webserver on your local machine, and spit back a link you can try in your browser. Ctrl-C to kill when you are done.

Sure, most browsers will open raw local files, but you run into issues with cross-site requests and over permissions. And the closer you can imitate a real webserver in development, the better.To install:
[sudo] npm install -g swank
It is my first item on npm. I was quite impressed with how easy it was to create a module and publish it. It is really nothing more than npm init (which is required for things like Heroku and good practice anyway) and npm publish.

I'd like to eventually drop the Connect middleware requirement, but it works for now (and it is realistically about 15 lines of code). Here is the project on npm. Let me know what you think!

EDIT: The latest version of swank now has optional ngrok support. Just add the --ngrok flag to the command, and have your server tunneled to the outside world. Pretty cool!

Tuesday, January 28, 2014

Temporary solution

I'm still working on getting an embedded car music player working, but I've been so busy with so many other things recently that I've rigged together a temporary solution. I got a spare wireless charger and literally hacked it into a car dock (using leftover plastic and hot glue - beautiful). Then some Tasker scripts turn on Bluetooth when wireless charging and automatically start playing music when connected to a Bluetooth audio receiver (this one is perfect and really cheap).


I had to put in a pretty long delay on connection before playing, thanks to some quirks with BT audio media volume levels since Jelly Bean. However it stops very quickly when removed or when the car shuts off. The audio quality is superb as well. No more risking my life trying to play some music in my car.

Friday, January 3, 2014

A little thing

I've been out of town for a couple days, so as soon as I got back I was desperate to build something. My wireless charger for my new Nexus 5 happened to be here, so on a whim I looked to see if Tasker was capable of telling if this was the case. Sure enough, Tasker can detect the charger flawlessly. So what to do with it...

I decided opening the desk clock was a little boring. I also noticed Tasker had a whole bunch of image manipulation methods and also an HTTP Get method. So I threw together some Ruby to scrape Flickr for a random image and pushed it to Heroku.

It seems that you can no longer easily export Tasker profiles, or else I would post it here. But it is pretty simple:
  1. Add a new Profile for Event > Power and set it to Wireless (or Any if you prefer)
  2. Create a new Task:
    1. Turn on Night Mode
    2. Set Variable %FLICKRIMG to Download/flickr.img (a static name will replace old images to save space)
    3. HTTP Get
      Output File: %FLICKRIMG
    4. Set Wallpaper to %FLICKRIMG
  3. Also create an Exit Task to turn Night Mode back off and set the wallpaper back to normal (or don't, and have a new wallpaper for the day!)
Here's a video of it in action. For a 3 hour project, I'm pretty excited.

Thursday, August 29, 2013 - Official Announcement

I finally have reached a point on that I feel comfortable announcing it to the world.

For those of you who don't know, for over a year (I think it has been closer to two), I've been tossing around the idea of creating a URL shortener for Wikipedia. The idea is that there are several social media platforms (Twitter, Facebook, Google+, SMS, etc.) where you cannot create hyperlinks with custom text. Instead, the platform recognizes URLs and formats them as links for you. Often (e.g. Twitter), you are limited by the number of characters you can have, so URL shorteners are popular. However, often these services give you a nonsense URL ( If you want to link to Wikipedia, and you want it to be clear that that is where it is going, you can use That way, you can use the URL inside your sentence, for example:
TIL that 's brother was a pretty cool guy.
The link to the Wikipedia article on Aldous Huxley is simply the name of the article on Wikipedia. I didn't have to generate this on the site (although I still could), I just knew the name of the article. Also notice the second link has an "a." at the beginning. This is a link I generated on the site. I told I wanted a URL that said "julian" to point to the article on Julian Huxley. For the time being, I'm calling this aliasing (but I'm looking for a more clear name).

The service allows for multiple articles for the same alias (that's where the "a." part comes from). If you try to go to an article that doesn't exist, it will take you to the search results page. It also handles languages relatively well. By default, it will detect your language and send you to the right version of Wikipedia. You can also force a particular language by generating it on the site. For now only a handful of languages are supported, but soon I will add all language versions of Wikipedia (and there are hundreds).

The base functionality has been working for a while (this was my first foray into databases, well before I knew SQL). Over the last few weeks, I've really been working on the UI and learning some fancy Javascript tricks in the process. The site uses jQuery (and jQuery-UI), and uses aAjax calls to PHP pages. You can see the source code on GitHub if you are curious. There is a bug with IE<=9 I still need to fix and two more features I plan to add (and I'm sure there are some Unicode bugs to still be worked out), but I think it is ready to be used.

Please, use it! Give me the satisfaction that I made something useful. :)

Tuesday, August 13, 2013

Discrete potentiometer

My car music player is finally coming along to the point where it looks like I might actually finish it. I will do a post about the software soon, and probably a more general post on the hardware later, but right now I want to talk about a knob I will be using on the device.

Your car stereo probably has a volume knob that spins all the way around. You can keep twisting it in either direction as much as you want, and the software will determine when you've topped- or bottomed-out. You probably didn't think too much about why this was the case, but there are some nice features that can be implemented thanks to it.

For one, you can have multiple volume controls. Often there are volume buttons on your steering wheel. A knob with set start and end positions means it is tied to a particular state, but changing the software state somewhere else doesn't change the physical state of the knob. Some car stereos (I know mine does, and I bet most do now) will set the volume to 0 when the car starts, and ramp it up gradually. That way, if a friend borrowed your car and cranked the volume up really high, you won't get a nasty surprise when you get in.

While most knobs are potentiometers with a continuous resistance from 0 to X, knobs like this have discrete states (they 'click' in small increments). To make one of these, I bought a cheap rotary switch. This is a SP12T (single pole, 12-throw) switch, so one input can be connected to any one of 12 outputs. Some rotary switches will let you twist all the way around forever in either direction, but this one did not, so I had to pry the case open and carefully cut out a little piece of plastic that was blocking the gap between positions 1 and 12. This particular switch had a small ball bearing and a spring, both of which I almost lost and I struggled for a few minutes to put everything back together (fair warning).

Now that the switch can spin freely, we need it to be able to turn a setting up or down. To know if the knob is spinning clockwise or counterclockwise (or not at all), we can simply compare the current state of the switch to the previous one. There are a few ways to read the switch state. Connecting every output to its own GPIO pin on your microprocessor would take up 12 pins which is a very inefficient use of your pins. One option would be to connect each pin to the first 12 inputs of a 16-to-4 encoder, which would encode your signal into a 4 bit number, taking up 4 GPIO pins. However, there is still a better way to do this.

By soldering resistors of equal magnitude between each of the positions, we've created what is essentially a discrete potentiometer. In this case, I used 100 ohm resistors, so this is a 11*100 = 1.1K ohm potentiometer (the first state is 0 for 0 ohms, so the last state is 11 for 1100 ohms), but one which can only take values 0, 100, 200, 300... Now we can connect it to an analog pin, and save our digital pins for other things.

There is still one issue here, however. The analogRead() function is going to give integer values from 0 to 1023. That means our values are going to be 0, 85, 171, 256, 341, 427, 512, 597, 683, 768, 853, 939. But these aren't exact. These resistors have a margin of error around 2% (for most resistors, it is around 5%, but I bought more accurate ones for this situation). The supply voltage can fluctuate as well. That means we need to allow for ranges, so maybe 0-42 means position 0, 85-128 means position 1, and so on.

A easy way to do this would be:
However, this is a very inefficient command for a microprocessor. First it multiplies two integers, then it divides two integers, creating a float (microprocessors like the Ardunio are very slow at doing floating point math, particularly division). Then it converts that back to an integer. This one line translates to 4 logical steps, which translate into a whole lot of steps for the processor.

The way I implemented this was not nearly as slick, but it runs about 10 times faster. If we look at our predicted analogRead() values in binary:
Notice the first four bits (highlighted in red) are unique for each value. That means that we can get a good idea which state we are in just looking at the 4 most significant bits. If you think about it in decimal: let's say you were going to get numbers between 0 and 1000, and you wanted to quickly tell if that number fell in the first 250 number bin (0 to 249), the second (250 to 499), and so on. You don't really care what the number in the one's place is in this situation. Just by looking at the hundred's and ten's places, you can tell which bin it's in (26X is in the 2nd bin, and 8XX is in the 4th). So back to the binary above, if we trash all the lower-significance bits (which might vary slightly), we can still get the right answer.

There is one little bit of weirdness here. Four bits means 16 possible values, 0000 to 1111. But we only have 12 positions we care about. Look at the binary version of the predicted value of position 3 (0100000000 or 256). If analogRead() returned something a little high, like 275, we would be okay, since the first four bits are still 0100. But what if it returned something a little low, like 245 (binary 0011110101)? This has a smaller value for its four most significant bits. However, notice also that we skipped from 0010 for position 2 to 0100 for position 3 (right over 0011). This is because 1024 is not divisible by 12. In hindsight, getting a 8- or 16- position rotary switch would have been a better plan. If we give both 0011 and 0100 to position 3, we've solved this problem. We have to do this as well for positions 7 and 9 (thanks to the fact that gcd(12,1024)=4). The shortest margin of error that would return a wrong value is then 21 (for example, if the value for position 2 was high by 21-- 192, it would be read as position 3). That is a hair over a 2% margin of error. 

Below is the code I used for this. It is not nearly as concise or elegant as the single line from before, but it is faster by an order of magnitude, as bitwise shifts are computationally cheap. If you have a better way to do this, I would love to hear your suggestion.

   int c = analogRead(0)>>6;
     case 0:
       return 0;
     case 1:
       return 1;
     case 2:
     case 3:
       return 2;
     case 4:
       return 3;
     case 5:
     case 6:
       return 4;
     case 7:
       return 5;
     case 8:
     case 9:
       return 6;
     case 10:
       return 7;
     case 11:
     case 12:
       return 8;
     case 13:
       return 9;
     case 14:
       return 10;
     case 15:
       return 11;

Sunday, May 12, 2013 API: genre influences

I've been talking about writing something using the API, and I finally have. Nothing to exciting here, just testing the water. This little script (using pylast) gets the top tags from your top artists, and scores them, giving you a percentage for each tag. It is pretty shitty code; this was a one-off thing, and the way it (I) can't make up it's (my) mind whether to use lists, dictionaries, or tuples is pretty embarrassing. Be sure to get an API key from first.

import pylast, math, operator, sys

# This is a simple script to play with pylast   #
# and the API. It goes through your     #
# library and gets the top tags for your top    #
# artists, and weights them based on playcounts,#
# giving you a percentage for different tags.   #
# Be sure to install pylast!                    #
#         #
#   Charles Knight -     #

usage = "python username artist_limit tag_limit [log_plays]"

if len(sys.argv) < 4:
 print usage

username = sys.argv[1]
artist_limit=int(sys.argv[2]) #How many artist's tags to use (e.g. top 50)
tag_limit=int(sys.argv[3])  #How many tags for each artist
if len(sys.argv) > 4:
 log_plays = 0  #1 to take the natural logarithm for playcounts. This smooths out your results by
    # giving more weight to artists with fewer listens (0 weighs by normal playcount)

# You have to have your own unique two values for API_KEY and API_SECRET
# Obtain yours from for

#This is an exclude list for tags. Here are some of the shitty ones I got. reducing tag_limit might help
bad_tags = ["female vocalists", "singer-songwriter", "rock opera", "80s", "90s", "political", "epic", "canadian", "megaman", "female vocalist", "eminem", "60s", "female fronted metal", "bass", "christian", "british"] 

all_tags = {}

# In order to perform a write operation you need to authenticate yourself
network = pylast.get_lastfm_network(api_key = API_KEY, api_secret = API_SECRET)

mylibrary = pylast.Library(user = username, network = network)

artists = mylibrary.get_artists(limit=artist_limit)

for a in artists:
 artist = a.item
 playcount = a.playcount
 if log_plays:
  playcount = math.log(playcount)
 name = artist.get_name()
 top_tags = artist.get_top_tags(limit=tag_limit)
 weights = []
 tags = []
 for t in top_tags:
  tt = t.item.get_name().lower()
  if tt not in bad_tags:
   weight = float(t.weight)
 sw = sum(weights)
 for i in range(len(weights)):
  weights[i]=weights[i] / sw
  tag = tags[i]
  if tag in all_tags:
   all_tags[tag] += weights[i]*playcount
   all_tags[tag] = weights[i]*playcount

#This black magic came from StackOverflow. Sorts a dictionary by value into tuples, no idea how. Requres 'operator'
scores=sorted(all_tags.iteritems(), key=operator.itemgetter(1))

ss = sum(s[1] for s in scores)
for t,s in scores:
 print '%-22s ==> %5s' % (t, str(round(s/ss,3)*100)+"%")

Tuesday, April 30, 2013

DSP + Arduino

Amanda Ghassaei, who has this amazing project I just found out about, also has a great Instructable for doing DSP with Arduino. A few hours and one blown op-amp later, I was able to get my Arduino to pickup my bass playing. It samples around 38.5 kHz, but I only had it send serial data back every 10 ms (a measly 100 Hz). Still, I was able to generate this graph with the data. You can see the clipping where I cranked it up to test the peak indicator LED.

The atmega328 has 32Kb flash memory. If I used half of that for a delay line, 16,000 samples / 38500 samples/sec = .42 second delay. I could get a longer delay by using external flash (e.g. SD card), but I worry that the read/write time would be low. It will be interesting to apply some of the concepts from class to actual signals. What does a 50-point averager do to my tone? (answer: probably reverb)

There's not a lot I can do with this now, since I can't hear the results of any processing yet. I just ordered a DAC0808 8-bit digital-analog converter. (The new DUE has two built-in DAC outputs, as well as PWM, although it still isn't an ideal platform for DSP). Eventually, I would like to get an FFT algorithm running and make a tuner. That will probably have to wait until after finals, though.

EDIT: Just found out about s.