MLOps on GCP Upcoming Local Shows Playlist (DataOps)

Storing Credentials in the Secret Manager

Now rather than hard code your credentials into the python script, we can utilize GCP's Secret Manager and then programatically access them in a secure manner. Navigate into the Secret Manager and click on + Create Secret. You can create a unique name for each of the credentials and enter their values ( no parenthesees ). You'll need at least the 4 below.

 search_spotify_tracks()

This function inputs an artist's name as a string and then tries, to search Spotify for artists with a similar name. I'm returning 5 matches, and will take the closest matching one later on. This function is wrapped around some rate limiting logic to handle the case if we reach Spotify's maximum rate limit. I told it to wait 1 minute and then try again and also to print out helpful statements which I'll be able to see in the logs. 

 The next part of this function goes through the results of the API request, first checking to see if there are in fact results to comb through. Next, it uses the partial_ratio() function from the fuzz package (formerly fuzzywuzzy), which performs partial fuzzy string matching between artist names. Basically it's trying to match the artist name from the SeatGeek API with the artist name from the Spotify API. It might be worth mentioning that the SeatGeek API actually contains the Spoitfy URI for each artist, so that might be a good candidate to use rather than string matching. Here I've opted to do the string matching as I was unsure of the coverage of the Spotify URIs and also I'd like to account for instances where bands are represented differently on each platform. For instance, on one API a artist might be labeled as Bruce Springsteen and on the other API might be labeled as Bruce Springsteen & the E-Street Band. Now what's really cool is that the fuzz package has actually be taken over and maintained by SeatGeek, which is prefect as they understand the need for this type of matching the best. They have a function called partial_ratio() that performs fuzzy string matching using Levenshtein distance metrics on sub-strings. So if the artists' name appear somewhere in the larger string they will receive a higher score. The function returns a score from 1-100.

You can find more on the fuzz package on their GitHub repo here: https://github.com/seatgeek/thefuzz

I'll then use that string matching on each of the 5 closest matches returned from Spotify, and only accept the artist if their fuzz score was greater than 80. After that I just extract the relevant info from the results. Most importantly I'll use the returned Spotify artist_id next to search for each artist's top 10 tracks.  

I'll use the retunred artist_id here with the artist_top_tracks() function from spotipy, which hits the Spotify endpoint that returns the top 10 tracks for each artist. After a quick check to see if the results returned are valid, I'll simply iterate through the top 10 track's, passing them into a sub-function get_track_audio_features() which I'll go over next. The data returned from this function contains audio features for each track as a dictionary. From there I'll add or update the artist info from before into that same dictionary. Then I can append this dictionary into an empty list. If there are no tracks returned I'll use a print statement to print the artist's name into the logs. 

 get_track_audio_features()

Short and sweet, this function calls the audio_features() function from spotipy, hitting the Spoitfy endpoint that returns the audio feature of each track. 

 get_secret_from_secret_manager()

This will be an important one to get right. Since I've stored the secrets in GCP's Secret Manager, I'll need a way to safely and programatically access them here in the script. Setting up and using the SecretManagerServiceClient() is pretty easy, but it expects the name of the secret to be returned in a very specific way - not simply the name of the secret. This function will access the secrets (after we've granted the permissions) and return them as a string. 

 Accessing the Secrets

Those we're all of my helper functions, now we can look at other parts of the script. First thing, I'll access those secrets and use them to build the parameters for each API. Now all of the handshakes have been taken care of and we can call any function from either API. 

 Getting Data from my Google Cloud Storage Bucket

Now we can use the Client() function from GCP's Cloud Storage library to retreive the csv that is stored there. It takes a few steps to get it from the bucket, to the blob, to a csv, then finally to a pandas dataframe.

 Pulling Data from the SeatGeek API

SeatGeek has a few different ways you can serach event's. The easiest here and most comprehensive would be to iterate through a list of abbreviated state pulling all future concert dates for each state one at a time. There are only 1001 cities in their database, but the results will contain concerts in nearby cities.  

 Expanding the Data for Multiple Artists per Show

Now each event returned has the main headliner as well as supporting acts. I'll expand the data to include a seperate row for each supporting artist as well. 

 Checking for Old Records

Now we don't need to bother the Spotify API for concert and audio track data if we already have it in our existing csv, so before we go making the API calls, I'll subset the dataframe into two; shows we already have concert and audio track data for and those we don't. It's common for touring artists to keep adding tour dates. We can check by using the id column from SeatGeek. 

 Pulling Data from the Spotify API

We've already checked that we're not using a dataframe consisting of concerts we already have data for, now we will also check to see if we already have audio track data for each artist in our existing dataframe. If we do, we will simply grab the data from our old records. If we don't then we will use the search_spotify_tracks() function to pull that data from the API for us.  If we can't find any data we'll print an error message to the logs. 

 Combine Records

Now we can combine all the data we have, old and new. Then drop duplicate records. Finally we will subset the data to be only future concert dates. This will help us stay in the Free Tier on GCP, but you could easily omit this to build a historical data set.

 Uploading to GCP Storage Bucket

The last step in out python script will be to upload and overwrite the exisiting csv stored in Cloud Storage. It's similar to how we read it in. 

 Setting Up an App Engine Service Account & Granting IAM Permissions

Next, we'll need to create an App Engine instance even though we're not ready to deploy anything just yet. The reason for this is that when you create an App Engine, you also create a service account which we will need to grant permissions for. Navigate to GCP's App Engine page, and click CREATE APPLICATION. Choose a location for you instance. Then inside of Identity and API Access, choose NEW SERVICE ACCOUNT.

 Enter a name and copy down the associated email address for the service account. 

 In the next step you'll be asked if you'd like to grant IAM premissions. It makes sense to do this now, but you can always naviagate to GCP's  IAM & Admin page and add these later on. As a matter of fact, after adding these, you should navigate there and double check that the permissions have been granted. The roles you'll need to grant are the following:

1. Secret Manager:  Secret Manager Viewer

2. Secret Manager:  Secret Manager Secret Accessor

3. Cloud Storage:  Storage Object Admin

4. Cloud Scheduler:  Cloud Scheduler Job Runner

 Setting Up Google Cloud SDK

Once you grant the permissions and GCP spins up your App Engine instance, you'll be asked to install Google Cloud SDK. Click that button and begin the install process. I also like to add it to my PATH, so that the commands are shorter and I won't have to typeout the path where the SDK is installed to run commands. 

You can find and follow the documentation here: https://cloud.google.com/sdk/docs/install-sdk 

 Deploying the App in the App Engine

Now all the components are in place and all that's left to do is push the shiny red button. 

Here's where all of the magic happens, in the terminal type:     gcloud app deploy  

Then choose:      Y  

That's it. The process should spin for a minute while it uploads the new versions of you files into your Cloud Storage Bucket. If you ever need ot make changes to the files simply rerun this command and your app will get updated in GCP. Another alternative would be to use something like GitHub Actions here to automate the build and deploy steps everytime you push changes into your GitHub repo - pretty cool stuff. 

Viewing the Logs

If the process ran successfully you can navigate to GCP's Logs Explorer page to view the process in real time. You'll be able to see warnings, errors, as well as any print statements you placed in your python file. Here's a sample output from the last run: