“Bach’s music… …is a really good resource for machine learning to learn music”
-Jacob Howcroft, Software Engineer, Google Doodle Team

I came across the Bach Google Doodle last spring:

It was created using a machine learning model which was trained on 306 of Bach’s chorale harmonizations. A chorale harmonization is a short piece written in four parts, typically a sung by a choir in a church. A typical example looks something like this:

The Google model was trained by taking a Bach Chorale, deleting random notes from it, and then having the model guess the missing notes. The basic way the Doodle works is that the user inputs a two-bar melody consisting of any notes they want. The Doodle then harmonizes the melody “in Bach’s distinctive style”.

After inputing a single-line melody of your choice into the model…

…the AI returns a four-part harmonization in the style of Bach:

What I find fascinating about this Doodle is not the high quality of the music that it produces (the beginning and end usually sound good, but things tend to go off the rails in the middle), but rather is the type of music that it produces. It doesn’t harmonize the melody simply by producing lead-sheet style chords. It actually produces four-part counterpoint.

What’s the difference between counterpoint and a lead sheet? A typical lead sheet looks something like this:

There’s the melody with the name of the chord written above each note. This lets the musicians know which notes they should play along with the melody as accompaniment. And this “lead sheet notation”, at least in my opinion, is a major achievement of Western Civilization. It allows an entire song to be condensed onto a single page, with all of the basic information intact.

But lead sheets leave a lot of room for interpretation by individual performers: the performers have a lot of freedom regarding the specific notes that they choose to play. This is in stark contrast with classical music, where all of the notes tend to be written out explicitly.

In other words, the Doodle could have “harmonized” the melody into a lead sheet, like this:

Technically, the chords are the same in both outputs. What makes the Bach Doodle more sophisticated is that rather than producing a single series of chord symbols, it outputs four distinct melodic lines. And this actually gives some insight into how modern Western music developed in the first place.

A common way to start learning music is to learn chords, maybe on a guitar or on a keyboard. And this is a great way to get started: there are countlessly many songs that only require three chords in order to play. For example, this Lynyrd Skynyrd classic:

But chords are deceptively simple. They are the Python of music. They’re easy to learn, they are very powerful, and they can do amazing things, but they’re actually a very recent historical phenomenon, the product of hundreds if not thousands of years of development.

Humans have been making music from time immemorial, but it’s really only since the European Middle Ages that there has been anything resembling modern polyphony. For example, if you listen to Peking Opera, no matter how many people are singing or playing instruments, they’re only ever playing one unique note at a time. (and where polyphony did exist in non-Western cultures, for example in Africa and Oceania, it was usually in the form of drones like on a bagpipe.)

How did polyphony develop in the West? For centuries, monks had sung single-voice Gregorian Chant. As time went on, another line would be added to enhance the harmony, often resulting in the kind of parallel motion you might remember from the monks in Monty Python and the Holy Grail:

Although this chant contains three separate lines, they are all perfect parallel copies of each other. In data science speak, there is perfect multicollinearity, with no linear independence whatsoever. But, as time went on, these additional voices became more and more independent, until by Bach’s time, they had developed into the modern tonal harmony that we know today.

So let’s take one more look at some four-part harmony, in the form of Beethoven’s famous Ode To Joy:

This particular arrangement includes the lead-sheet chord symbols. But if you look closely, you’ll see that there’s a lot going on under the surface. The first five notes, for example, are written as “F” using chord symbols, yet in reality there are four distinct lines of music, each with its own individual contour and register that can’t be captured with only a chord symbol.

As modern musicians, it can be easy to forget that there is more to tonal harmony than chords, just as it can be easy to forget, as data science students, that there’s more to computer science than high-level programming languages like Python. Chords are so simple to learn and easy to use that one can forget that they actually developed out of something more fundamental.

Google’s Bach Doodle is a great tool for getting back in touch with where chords and, more broadly, Western tonal harmony actually came from: multiple lines of music playing at the same time, namely, counterpoint.

“Good work!”

— Abhishek, 12/13/2019

“It’s a bad model”

— Abhishek, 12/16/2019

For last week’s project at the Flatiron School, my partner Dayana and I investigated the effect of bad weather on transportation patterns in Chicago. We used hourly weather data to create a dummy variable to represent “bad weather” on a given day, and then tested our null hypothesis that daily CTA ridership during bad weather is the same as CTA ridership during good weather.

Lo and behold, the p-value was less than 0.01:

In other words, we found that bad weather has a statistically significant effect on CTA ridership, which is exactly what we had set out to prove. Abhishek congratulated us on a job well done, and I was happy to have drawn a tangible conclusion using real-world data. I was so inspired that I decided to download another data set from the Chicago Data Portal, this one about number of car crashes per day, and see if there was any significant correlation there.

And it turns out there was. When I ran another test, this time using the null hypothesis that bad weather has no effect on car crashes, there once again was a statistically significant result, with a p-value less than 0.005:

I was feeling good about hypothesis testing.

Then we started learning linear regression. I had thought that linear regression and hypothesis testing were just two ways of doing the same thing. So with this in mind, I put the same data I had used in my car crash hypothesis test into an OLS regression, and this was the result:

Once again, the p-value is significant, and I thought that this was the end of the story. However, Abhishek was walking by as I ran the test, and without even being asked a question, he pointed to the 0.012 R-squared value and simply said, “that’s a bad model”.

This confused me. It was the same data, and it had produced just as good of a p-value as the hypothesis test had. I didn’t realize that it was possible to have a low p-value and a low R-squared value at the same time.

Fortunately, a quick Google search clearly illustrates this phenomenon of two distributions with similar coefficients and p-values, but with very different R-squared values:

The problem became much more obvious when I visualized the regression using Seaborn:

It turns out that a fundamental assumption of linear regression is that there is a linear relationship between the independent variable and the dependent variable. That is the data points should be as close as possible to the regression line, not on completely opposite ends of the graph as they are here.

That being said, the other three assumptions of linear regression didn’t seem to be violated at all:

• 1) A qq-plot shows that the residuals are reasonably normally distributed. The entire distribution of car crash data is remarkably similar to a normal distribution, too, both in terms of skewness and in terms of kurtosis:

• 2) A Breusch Pagan test shows that we can’t reject the null hypothesis of homoscedasticity:

• 3) There is no multicollinearity, since there is only one independent variable.

Divide and Conquer

So regressing on a binary variable wasn’t going to work. But occurred to me that the daily “bad weather” categorical variable didn’t need to be binary. It had come from hourly weather data, and so it could be split into 25 different categories, each representing a number of hours in a day ranging from 0 to 24. I created a new ‘hourly bad weather’ variable, ran the regression again, and managed to raise the 0.012 R-squared value to a whopping 0.019.

Long story short: bad weather, while having a significant effect on number of car crashes, accounts for less than 2% of the variation in number of crashes per day. If I wanted to gain any insight into the other 98%, I’d need to think of something else.

Time to Give Up?

I was ready to give up, but I had one last idea. Since I now had data for 22 unique “hours of bad weather in a day” values, I decided to take the average of each of them, and regress ‘total crashes’ on the result. The result was as follows:

This yields an R-squared value of .304. It’s better, but at this point the model is pretty much useless. All it says is that, taking yearly aggregates, one extra hour of bad weather on a given day explains 30.4% of the 1.17-car increase in crashes.The other 69.6% is explained by something else.

But this is not surprising. There are so many factors involved in car crashes, it’s not surprising that my crude “bad weather” metric can only account for 30% of the variation in aggregate crash data. For example:

• My “bad weather” variable is not a consistent metric. “Rain” one hour isn’t identical to “rain” (or “snow”) another hour
• Car crashes are not completely independent events (e.g. multiple-car pileups)
• My “car crash” variable doesn’t take into account the severity of the crash
• Many crashes go unreported

Fun Facts

Since by now I had all of my formulas and models set up, I decided to run a few more tests, just to see if I could find anything interesting, or at least anything with an R-squared value higher than 0.019.

Crashes vs Temperature

There is an R-squared value of 0.039 (which at least is higher than 0.019) when regressing car crashes on near-freezing temperatures (temperatures between -5°C and 5°C). I thought this made sense. These are the temperatures that roads are beginning to freeze, are not salted, and drivers are not expecting the roads to be icy.

But if you look closely, the coefficient is negative, meaning that there are significantly fewer crashes on icy days, not more. One possibility for this could be that the crashes that occur on icy days are more serious. So I created a “serious crash” variable, which denoted crashes that resulted in at least one injury. This did not change the results, however. What did end up having more of a significant correlation was regressing number of serious crashes on temperature (in °C). This produced an R-squared value of 0.183:

Days of the Week

• There’s a 14.6% increase in car crashes on Friday:

• There’s a 16.6% decrease in car crashes on Sunday:

Both are highly statistically significant, but explain less than 17% of the variation in number of car crashes.

Conclusion

The purpose of hypothesis testing is to determine the validity of a particular claim. The purpose of linear regression, by contrast, is to determine the specific relationship between a dependent variable and one or more independent variables. Linear regression requires certain conditions to be met before any conclusions can be drawn. As such, it is not always appropriate to run a linear regression using data from a hypothesis test.

“Isn’t that the game that old ladies like to play?” Whenever I ask someone if they play Bridge, this seems to be the response. And it’s a reasonable one: according to News@Northeastern, the average age of a competitive bridge player in 2018 was 71, and the infamous Bridge Murder case, a rough proxy for the peak of the game’s popularity, occurred in 1929.

Growing up in Michigan, I didn’t play bridge at all. I did, however, play lots of Euchre, and when I finally learned to play Bridge in college, I was struck by how similar the two games were. Part of what interests me about data science is the potential for developing software that can learn and play different games, and I think that both Euchre and Bridge would be prime candidates to use for learning some of the relevant concepts. What follows is a brief overview of the similarities and differences between the two games.

Basic Set-Up

Both games have the same basic format:

• Four players sit around a table with partners facing each other
• The trump suit is determined
• A card is led
• Play continues clockwise
• Each position must follow suit, if possible
• The winner of each trick leads the next trick
• Each partnership attempts to take as many tricks as possible
• Points are awarded based on how many tricks are taken
• Each hand consists of three phases: bidding, playing, and scoring

The Bidding Phase (a.k.a. the “Auction”)

The process of determining the trump suit is commonly referred to as “bidding”. In Euchre, there are 7 words that you are allowed to use during the bidding process, while in Bridge there are 15:

What makes bidding in Bridge more complicated than in Euchre, however, is not the number of permitted words, but the purpose of the bidding itself. In Euchre, the sole purpose of bidding is to determine the trump suit, and as a result, there can only ever be one “non-pass” bid. As soon as anyone says anything other than “pass”, the bidding is over and it’s time to start playing the hand.

Here are some examples of bidding in Euchre:

In Bridge, the purpose of bidding is still to determine the trump suit but is also to determine how many tricks the declarers believe they can take. As a result, in Bridge, the bidding goes on as long as it needs to, the only rule being that each bid must be higher than the bid that came before it.

Here are some examples of bidding in Bridge:

The number next to the suit is the number of tricks the bidder expects to be able to take above six (six tricks is called a “book”).

Since each bid must be higher than the bid before it, this means that the suits have a ranking: Spades (♠) is highest, followed by Hearts (♥), Diamonds(♦), and finally Clubs(♣). This can be seen in the above example, in which the bid of 3♥ is higher than the bid of 3♦ (although it’s still lower than the bid of 4♣). The fact that spades is the highest suit is reflected in the common expression “to have something in spades.”

For reasons that will be discussed later in this post, a contract in a major suit (♠ or ♥) is preferable to a contract in a minor suit (♦ or ♣). As such, most modern bidding conventions follow the “Five-Card Majors” rule, meaning that when one opens the bidding in a major suit, one is guaranteeing at least five cards in that suit.

Evaluating Your Hand

Before one can bid for trump, one must determine whether one’s hand is strong enough to bid at all, or if one should just say “pass”. The now-universally-accepted method of evaluating a bridge hand was developed by Charles Goren in the 1930s and is called the Goren Point Count System. It assigns High-Card Points (HCPs) as follows:

• Ace: 4 HCPs
• King: 3 HCPs
• Queen: 2HCPs
• Jack: 1 HCP

So to practice, how many HCPs does this hand contain?

How about this one?

How many HCPs are contained in the entire 52-card deck?

How Many Points Do I Need to Start Bidding?

Now that we know how to count our High Card Points, here are some rough guidelines for bidding in Bridge:

• 13+ HCP are needed to open
• 6+ HCP are needed to respond to partner’s open with a different suit at the 1-level
• 6+ HCP are needed to raise your partner’s opening suit to the 2-level
• 10+ HCP needed to respond to partner’s open with a different suit at the 2-level
• 10+ HCP are needed to overcall an opponent’s open
• ≤5 HCP: just say “pass”

As with Euchre, the most commonly-heard word during the bidding phase is “pass”. This is because 13 HCPs are required to open, but an average hand contains only 10. Let’s look at an example:

What should South do in this situation?

Who will open the bidding?

Then what will happen?

If everyone follows standard bidding conventions, the bidding will probably look something like this:

Neither South nor West have enough points to open. North has 14 HCPs and a five-card major, so he opens one Heart. East has 11 HCPs and a five-card suit, enough to overcall with Clubs at the two-level. South has a very good fit in Hearts, and bids accordingly. North ends up as declarer, with South as the dummy. It will be up to North to use the cards in his hand and the cards in the dummy to take ten tricks.

Bridge bidding can get much more complicated than this, particularly when partners have extremely good hands and stand to take 12 or possibly even all 13 tricks. But this is enough detail for an introductory post. In any event, no matter how complex one’s bidding conventions are, the end result will always be the same thing: the trump suit is declared, and then the hand is ready to be played.

Playing The Hand

Playing a Bridge hand is essentially the same as playing a Euchre hand: one person leads and everyone follows suit, the object being to take as many tricks as possible. It’s actually simpler in the sense that there are no Bauers: a jack is just another card, no matter the suit.

One difference from Euchre is that the person to the left of the declarer (the person who declared the trump suit) always leads the first trick, unlike in Euchre where the person who leads is always the person to left of the dealer.

Another difference is that, in Bridge, the declarer plays both his and his partner’s hand. After the first card is led, the declarer’s partner lays his cards down on the table for everyone to see, and his position is referred to as the “dummy”. It looks something like this:

Otherwise, the play is exactly the same: someone leads a card, everyone must follow suit if possible, and the winner of one trick leads the next trick.

One practical advantage of always having a dummy is that everyone gets fairly regular breaks. This is one of the reasons bridge games can go on for hours and hours without ever stopping.

Scoring

Scoring in Euchre is simple. If the declarers take three or more tricks, then they are awarded 1, 2, or 4 points, depending on the number of tricks taken and whether or not the declarer went alone.

Scoring in Bridge is more complicated, particularly if you’re playing Rubber Bridge, in which long-term considerations need to be taken into account. That being said, basic scoring in Bridge is relatively straightforward:

If the contract is made:

• 20 points per trick are awarded in a minor-suit contract,
• 30 points per trick are awarded in a major-suit contract,
• 40 points for the first trick and 30 points for each additional trick are awarded in a no-trump contract,

If the contract is not made:

• A 50-point penalty is assessed for each trick that the declarer failed to make.

A large bonus is also awarded if a contract worth at least 100 points is made. This creates a big incentive to bid for one of the following contracts: 3NT, 4♠, 4♥, 5♦, 5♣.

Final Thoughts

Although Bridge is a difficult game to master, the basic rules are actually very simple: as long as you remember the order of the suits, you will never make an illegal bid, and as long as you follow suit while playing you will never play an illegal card. Once you’ve grasped those concepts, getting better is just a matter of gaining more experience and making sure that you and your partner are following the same conventions.

Happy card-playing!

This is an example post, originally published as part of Blogging University. Enroll in one of our ten programs, and start your blog right.

You’re going to publish a post today. Don’t worry about how your blog looks. Don’t worry if you haven’t given it a name yet, or you’re feeling overwhelmed. Just click the “New Post” button, and tell us why you’re here.

Why do this?

• Because it gives new readers context. What are you about? Why should they read your blog?
• Because it will help you focus you own ideas about your blog and what you’d like to do with it.

The post can be short or long, a personal intro to your life or a bloggy mission statement, a manifesto for the future or a simple outline of your the types of things you hope to publish.

To help you get started, here are a few questions:

• Why are you blogging publicly, rather than keeping a personal journal?
• What topics do you think you’ll write about?
• Who would you love to connect with via your blog?
• If you blog successfully throughout the next year, what would you hope to have accomplished?

You’re not locked into any of this; one of the wonderful things about blogs is how they constantly evolve as we learn, grow, and interact with one another — but it’s good to know where and why you started, and articulating your goals may just give you a few other post ideas.

Can’t think how to get started? Just write the first thing that pops into your head. Anne Lamott, author of a book on writing we love, says that you need to give yourself permission to write a “crappy first draft”. Anne makes a great point — just start writing, and worry about editing it later.

When you’re ready to publish, give your post three to five tags that describe your blog’s focus — writing, photography, fiction, parenting, food, cars, movies, sports, whatever. These tags will help others who care about your topics find you in the Reader. Make sure one of the tags is “zerotohero,” so other new bloggers can find you, too.