Claudia’s Blog

End of the Term

2022-07-18T00:00:00+00:00

Claudia Donahue 2022-07-18

Using R for Data Science: My Thoughts

I have really enjoyed working in R and using RStudio this term, after not having used it for several years. A lot of what I use in the future is going to depend on what I have access to in the workplace. I know there is a great chance I will be able to use R, so I expect to have a lot of opportunity to use it. If I had to make a prediction, I will probably just be battling the status quo of using Excel to handle anything related to data.

It will be helpful that I now have experience using R Markdown to create .pdf and .html files to share reproducible work.

What Has Changed for Me

Now that I have this recency and depth in R, I will have to decide when I feel it’s appropriate to use R or Python, or another tool (probably Excel) at work. I also feel more equipped to work consistently on side projects or further self study outside of work, and I feel fortunate to know some experts at NC State, if I ever need to reach out.

My Next Exploration

I am interested in trying out a competition, like Kaggle, or one through my employer. I also already have a request from a friend in my neighborhood to help them with some modeling. I am looking forward to seeing how I can help!

At work, I am sure I’ll be exploring how to efficiently create dashboards useful for leaders in my organization.

In my community, I like the idea of presenting publicly available data in ways others can better understand and draw conclusions.

I am looking forward to completing my final project Shiny App with multiple pages and brainstorming similar things I could create as community tools.

Machine Learning Module

2022-07-14T00:00:00+00:00

Claudia Donahue 2022-07-14

Learning About Machine Learning

We just finished the course’s section on Machine Learning. We did a little bit of unsupervised learning, but mostly supervised learning, meaning we were trying to predict a target variable, versus just looking for patterns and relationships in the data.

Supervised Learning

We covered the standard regression model and then generalized linear models to predict (or classify) a target variable. Then we worked on nonlinear methods and ensemble learning.

Unsupervised Learning

Finally, we looked at two methods of unsupervised learning, principal components analysis and clustering.

What I Liked Most

In this module, I would say the classification trees were most interesting to me. A commonly used example seems to be using the data containing characteristics of passengers on the Titanic to predict whether each one survived the ship sinking. I like that you can output a decision tree, and then follow along that tree to make a prediction on an observation. I think that sort of visualization alone can be quite useful in highlighting feature importance.

I liked using the caret package because it simplified training a huge variety of models. Here we will fit a model using caret::train, and then display a visualization of the resulting tree to show how cool it is as a tool to understand the importance of different variables. We are predicting survival based on variables like age, passenger class, ticket fare, and sex:

library(caret)
fit <- train(x = training[,2:8], 
                y = training[,1],
                method = "rpart",
                preProcess = c("center", "scale"),
                cp = 0.001,
                trControl = trainControl(method = "cv", 
                                         number = 3)
                )
# Plot the tree using the rattle package
library(rattle)
fancyRpartPlot(fit$finalModel)

You could take your own characteristics, for example, and follow through this tree to get a prediction on whether you would have survived the sinking of the Titanic. If you are ‘male’, it is pretty simple. 81 percent of males passengers died. If you are a female, the models predicts your survival only if you had a higher Pclass ( passenger class) or paid more for your passenger fare.

More on Why I Like Classification Trees

As I begin working on my final projects for both my summer courses, two great datasets I’m eyeing are:

one from the NTSB (National Transportation Safety Board) on airplane crashes and fatalities. I like to fly, and potentially may buy a small plane at some point. I want to pick one that not only is affordable for me, but also that would keep my family and me alive in the event of an mishap.
a second on vehicles colliding with cyclists and pedestrians in North Carolina. I am hopeful that research can lead to evidence-based improvements in my local community, like more funding and increased safety measures for alternative transportation.

A decision tree with this sort of data may enable a community leader to plug in locally relevant observations (themselves/their neighborhoods/their grandkids) and understand the likelihood of bike/pedestrian fatalities if nothing changes. They could get an idea where in the community bike/pedestrian funding would make the largest impact.

Hopefully I can use the skills I’m gaining this summer to make a positive impact!

Automating R Markdown

2022-07-08T00:00:00+00:00

Claudia Donahue 2022-07-08

The Assignment

Our second project in this summer course was to use this data to make several models to predict the number of times readers shared an article from Mashable. The data set was split into six channels–lifestyle, entertainment, business, social media, tech, and world–and our job was to automate a report for each channel from a single R Markdown file. We were also practicing building, testing, and comparing regression and ensemble models, using the caret package in R.

Our Solution

You can see how our resulting six reports at this link, and delve deeper into how we solved the problem at our Github repo here.

My Thoughts

Happily, I do not think I would do a lot differently. I am glad to say we started working on this project as early as we could, and we had plenty of time to complete the work. I did notice that our predictive models were not making great predictions, so I might change the goal from predicting the dependent variable (number of shares) to predicting the logarithm of the number of shares.

The most difficult part I encountered was in building the random forest model. I initially made the model so complex–both the tuning parameters and the repeated cross-validation–that the caret::train() function would run and run, never finishing. Once I simplified the model by reducing the mtry tuning parameter and no longer repeating the cross-validation, the model worked.

The Takeaway

R Markdown is useful for so many reasons, and automating reports was within my reach. I think understanding this concept could be useful for me in the future.

Thanks for reading!

Update on How R Programming Is Going

2022-06-27T00:00:00+00:00

Claudia Donahue 2022-06-27

So Far This Summer…

We are just about halfway through the summer term (my final term in my program!), and I have made plenty of progress. We are done learning about how to program in R and the rest of the coursework will explore other skills.

The coolest thing I have learned so far in the course is how to share my work using R markdown and Github Pages. Before this course, I had a brief introduction to programming in R about 4 years ago and had not used it much since then. Markdown was not a big part of that weeklong intro for me. I have found it really useful to create something I could then pass along to a colleague who wants to know how I got my results or how to replicate my work with their data.

Another thing I am happy to have learned in R is how to create custom functions. This skill has been useful to develop so far my degree program when I want to use an algorithmic method to solve certain types of problems, but until now I have used other languages, like Python or Julia.

Here’s an example of a custom function that, given a numeric vector and a mean value to compare against, calculates the test statistic:

tStat <- function(vec, mean){
  
  tObs <- (mean(vec) - mean) / (sd(vec)/sqrt(length(vec)))
  
  return(tObs)
}

I look forward to learning more skills that will be useful in my career in operations research!

API Interaction: A Vignette

2022-06-26T00:00:00+00:00

Claudia Donahue 2022-06-26

The Vignette Assignment

Our class was assigned to choose an API from a list and create a vignette, demonstrating to a user how to interact with that API. I initially chose the Pokemon API but had very limited success working with it and switched after a couple (wasted?) days. I ended up using a COVID-19 data API.

I created a bunch of functions which allow a user to interact with the API and select options, like a country or timeline of interest. The user then gets a dataframe or a graph, depending which function. Then I explored the data further, using techniques we have been learning this term.

Reflections

I found that I was left with a lot of questions. It was interesting to note that the United States has a much higher confirmed case count than any other country, but I wondered about record keeping and reporting. The same with the numbers of deaths. I also learned that a lot of the compelling data is behind a subscription fee, as I wished I had access to population and GDP data within the API, but I did not want to pay for it and did not have an excess of time to figure out how to match it up.

I had trouble with this data set because it was not easy to categorize and explore the data the way we have been in other assignments. I had to add a new variable so that I had some interesting categories to compare across. Still, it was better than the Pokemon API, where I could not make a function return a decent dataframe in a reasonable amount of working time.

I am happy to be getting more comfortable using github for version control. Here is a link to my repository for this project.

Next time

Next time I have a project like this, I like to think I will start earlier and ask for help (probably) to make sure I can properly use the data I want to use (Pokemon – to get my kids interested in data science). I also probably will not choose an API just based on whether I have to sign up for my own key. Who knows–I might even pay my own money for access to an API one day!

Last thing

Here is the link to my github pages repo for this project. I made it public on the due date for the project, as directed.

Programming Background

2022-06-06T00:00:00+00:00

Claudia Donahue 2022-06-06

My Experience So Far

The majority of my programming experience was in the Fall 2021 term in ISE 535 Intro to Python. I also did a weeklong course, Intro to R Programming in 2018, and I took an introductory course in SAS during the summer of 2021. Unfortunately, at work I usually use Excel, and my programming gains atrophy quickly.

My thoughts on R so far in this course: I really like using Markdown in R to create a readable document. I think that this tool/skill will help me use R at work to create products for my coworkers who may not know anything about R or have any interest. And also for those who do, so they could understand and reproduce my work.

I also like using RStudio with Github so far, for version control. I have had trouble working on projects with partners/groups because we are all working on one file and can’t figure out how to best merge changes, so one team member does most of the work.

I don’t think I will miss much about other programming languages, especially Python since I am using it regularly in another course this summer! I do not consider R any more or less difficult than the other programming languages I have experienced, and it is much easier than foreign languages I have tried to learn, specifically Russian and French.

R Markdown Output

Below, I am going to plot the Cars data set to make sure I can get the image to show up on my blog post:

plot(cars, xlab = "Speed (mph)", ylab = "Stopping distance (ft)",
    las = 1, xlim = c(0, 25))

This is my first blog post

2022-05-19T00:00:00+00:00

My Quick Take on the Role of Data Scientists

A data scientist must be familiar with all aspects of their organization, aware of the data it has, and able to communicate with stakeholders clearly to inform decision making. They must understand how to employ and how to value the tools at their disposal, whether the tool is a team member who can program or an application simplifying machine learning.

But to really hold the title of data scientist and be reliable and essential within one’s organization, I think it takes more than the traditional Venn diagram of math, computer science, and subject matter expertise. I think this visualization, and several job listings cited in the blog posts we read for this assignment, were missing organizational trust.

I think any employer giving an employee broad access to its data must be aware of the risks inherent in sharing this information. Whether it’s private medical details, financial transactions, strategic business plans, or military troop numbers, leaders have much at stake within the data they collect.

Data Science vs. Statistics

With apologies for my limited understanding of what statisticians do, I think we could fairly state that a data scientist is a statistician with a particular set of skills, including some programming. Kind of a special flavor of statistician! I do not believe, based on the articles I was assigned to read, as well as my own set of experiences, that a statistician necessarily has to be capable of data science. Computer programming and a level of expertise in a subject other than stats are not requirements to be a statistician, but are necessary for a successful data scientist. Perhaps a data scientist does not always need to have as great an understanding of statistics as someone serving in a statistician role. But a strong understanding of statistics certainly would make a stronger data scientist, able to more confidently find and explore questions and solutions through data.

Me

My perspective is, of course, formed through my own experiences. I don’t fit neatly into the data scientist or statistician bin at all. My background includes forays into military rotary wing aviation, leadership, economics (undergraduate), and (lately) operations research. (Side note: Operations research is another interdisciplinary field with a really squishy definition)

I find many of the hard skills I have studied in these fields are incredibly perishable and also time consuming. The impact of those qualities is that moving to a new field can prevent a person from considering themselves a member of a previous field, even when the areas seem complementary.

For this assignment, we students read through multiple articles/blog posts which sought to define a Data Scientist and distinguish it from similar career fields. Multiple authors cited the private sector demand for these employees; however, I noticed no one touched on the need for data science within government. But that is my only experience so far. I look forward to gaining more perspective this summer through the teaching and fellowship in this course.

Blog Post Title From First Header

2021-03-08T00:00:00+00:00

Due to a plugin called jekyll-titles-from-headings which is supported by GitHub Pages by default. The above header (in the markdown file) will be automatically used as the pages title.

If the file does not start with a header, then the post title will be derived from the filename.

This is a sample blog post. You can talk about all sorts of fun things here.

This is a header

Some T-SQL Code

SELECT This, [Is], A, Code, Block -- Using SSMS style syntax highlighting
    , REVERSE('abc')
FROM dbo.SomeTable s
    CROSS JOIN dbo.OtherTable o;

Some PowerShell Code

Write-Host "This is a powershell Code block";

# There are many other languages you can use, but the style has to be loaded first

ForEach ($thing in $things) {
    Write-Output "It highlights it using the GitHub style"
}