Blog | Drew Skwiers-Koballa

Code analysis with SQL projects

2025-01-22T00:00:00+00:00

⏸️ Before you get started

In a previous article, “SQL database projects intro”, we created a SQL project with 2 tables and a stored procedure.

dbo.Product (table)
dbo.ProductOrder (table)
dbo.GetProductInfo (stored procedure)

SQL database projects are built on top of the .NET SDK - you don’t need to know .NET development to use them - but you do need the .NET 8 or higher SDK installed.

Want to quickly check if you have the .NET SDK installed?

dotnet --list-sdks

▶️ With the .NET SDK installed and a sample project created like the one from the intro article, you’re ready to go.

📝 Create a new view

Add a new view [dbo].[All Products] in a .sql file (dbo.AllProducts.sql) in a new folder Views under project folder.

CREATE VIEW [dbo].[All Products] AS
SELECT *
FROM dbo.Product

Build the SQL project, verifying the object references and our SQL syntax is correct.

dotnet build

🛠️ Check our code quality

While SELECT * seems like a nice shortcut to reference all potential columns, it comes with risks of schema drift, inadvertently clogging up I/O with extra data, and more. In general - your code is more fragile in ways that aren’t easily observed.

This is where SQL code analysis comes in handy, because it will check the database model created from our SQL project for detectable issues. We can run code analysis on our SQL project as a 1-off analysis with an extra property on the project build /p:RunSqlCodeAnalysis=true.

dotnet build /p:RunSqlCodeAnalysis=true

Now instead of the build succeeding with no warnings, we have two warnings:

warning SR0011: Microsoft.Rules.Data : Object name(All Products) contains special characters.
warning SR0001 Microsoft.Rules.Data : The shape of the result set produced by a SELECT * statement will change if the underlying table or view structure changes.

Getting ongoing analysis of our SQL code quality provides a quick feedback loop even before we send the code into a git repository. We can enable code analysis to run by default on every project build by incorporating that property into the project file itself with <RunSqlCodeAnalysis>. A sample project file with the code analysis setting added would look like:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build">
  <Sdk Name="Microsoft.Build.Sql" Version="1.0.0-rc1" />
  <PropertyGroup>
    <Name>ProductsTutorial</Name
<DSP>Microsoft.Data.Tools.Schema.Sql.Sql160DatabaseSchemaProvider</DSP>
    <ModelCollation>1033, CI</ModelCollation>
    <RunSqlCodeAnalysis>true</RunSqlCodeAnalysis>
  </PropertyGroup>
</Project>

🚧 Choose errors and warnings

Some code issues are worse than others - they might have a high impact or be difficult to change in the future. For these kinds of issues we can elevate the warning to an error that will cause our build to fail.

In our current project we’re getting a warning that the object name [dbo].[All Products] contains a special character (the space). Before I fix this issue I want to make sure that these kinds of mistakes don’t get ignored in the future.

The project property <SqlCodeAnalysisRules> can be used to exclude a rule completely or elevate a rule from a warning to an error. When choosing errors and warnings, the project property is a semicolon separated list with three settings available for each rule:

elevate a rule from warning to error with +!
no entry for the rule results in the rule throwing a warning
demote a rule to no warning with - (disable)

The setting per rule is prefixed to the rule name. Let’s turn the object name warning into an error message since names can be more difficult to change down the road. Right below the property <RunSqlCodeAnalysis> in our project file we add:

<SqlCodeAnalysisRules>+!Microsoft.Rules.Data.SR0011;</SqlCodeAnalysisRules>

If we build our project again using dotnet build we recieve one error and one warning:

error SR0011: Microsoft.Rules.Data : Object name(All Products) contains special characters.
warning SR0001 Microsoft.Rules.Data : The shape of the result set produced by a SELECT * statement will change if the underlying table or view structure changes.

⚠️ Note that when the build fails, the build artifact (.dacpac file) isn’t updated. You need a successful build for your database model to be ready for deployment.

✏️ Fix the view name

Using the information in the error message, including the filename and line number, we’ll fix the build error resulting from our modification to the code analysis warnings and errors. We should edit the view definition in dbo.AllProducts.sql to have a name without spaces, like [dbo].[AllProducts].

Once the view name is fixed, verify that your project now builds with a single warning message for SR0001 (SELECT *).

📍 Supress a specific code analysis finding

Code analysis rules provide a structure for evaluating your code, but at the end of the day its your project and you may have a reason for contradicting best practices. Instead of completely disabling a rule with the <SqlCodeAnalysisRules> project property, we can use the StaticCodeAnalysis.SuppressMessages.xml file to suppress a specific rule for a specific file.

Create a new file next to the SQL project file named StaticCodeAnalysis.SuppressMessages.xml. In this file we add an entry for the view file and the SELECT * rule.

<?xml version="1.0" encoding="utf-8" ?>
<StaticCodeAnalysis version="2" xmlns="urn:Microsoft.Data.Tools.Schema.StaticCodeAnalysis">
  <SuppressedFile FilePath="Views/dbo.AllProducts.sql">
    <SuppressedRule Category="Microsoft.Rules.Data" RuleId="SR0001" />
  </SuppressedFile>
</StaticCodeAnalysis>

If we use SELECT * in other files of our project the warning will appear for those instances, but we’ve specified that SELECT * is allowed in our view dbo.AllProducts.

Running dotnet build with this message suppression XML file in the project will result in no warnings being shown for our project now.

🧩 (optional) Incorporate additional code analysis rules

The small set of code analysis rules included with SQL projects are helpful, but there are quite a few community projects extending the code analysis rules. Incorporating these custom code analysis rulesets into your SQL project code analysis is quickly accomplished through package references.

A package reference is an entry in the SQL project file to a NuGet feed, like the community TSQLSmells on NuGet.org - https://www.nuget.org/packages/ErikEJ.DacFX.TSQLSmellSCA

We don’t even need to manualy edit the .sqlproj because the .NET tooling has a terminal command to add packages:

dotnet add package ErikEJ.DacFX.TSQLSmellSCA --version 1.1.2

When we run dotnet build again, the build succeeds but we now have two new warnings!

warning SML005: Smells : Avoid use of ‘Select *’
warning SML030: Smells : Include SET NOCOUNT ON inside stored procedures

While adding the custom rules to your SQL project takes just a few moments, the developers that work on the rulesets have invested countless hours of their time and share with the community. Remember to contribute to the projects that benefit your work as you are able.

🚀 (optional) Add a GitHub continuous integration (CI) pipeline

If we’ve pushed our project to a GitHub repository, we can utilize GitHub Actions to check our project for code analysis findings. This is a great way to automate watching for code smells in databases.

The pipeline definitions are stored in a .github/workflows folder. Create a new file code-analysis.yml in the .github/workflows folder. An example pipeline definition is shown below to get you started, but your branching strategy and code review process may vary.

# .github/workflows/code-analysis.yml
name: Run code analysis on SQL project
on:
  pull_request:
    branches:
    - main

jobs:
  build-with-analysis:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - name: Setup .NET
      uses: actions/setup-dotnet@v4
      with:
        dotnet-version: 8.0.x

    - name: Build with code analysis
      run: dotnet build -warnaserror ProductsTutorial.sqlproj /p:RunSqlCodeAnalysis=true

In this workflow we’ve added yet another flag to dotnet build so that all code analysis warnings are elevated to errors. This is useful when you are using code analysis as an optional check. If you’re using code analysis as a required check (highly suggested) selecting specific rules to throw errors instead via the SqlCodeAnalysisRules project property is a better approach.

Guess what!? Your code analysis CI check is also doing the basic model validation for the database - ensuring you’re not referencing objects that don’t exist and the SQL syntax is valid. That’s a win-win.

📚 Learn more

The tooling ecosystem around SQL projects includes VS Code, Visual Studio, and the SqlPackage CLI. Here are some additional resources to learn more about SQL projects for development and deployment:

Introduction to SQL projects

2024-12-13T00:00:00+00:00

SQL projects are a development capability for SQL Server and Azure SQL where your database objects are stored as code with integrations for CI/CD capabilities like code quality checks and dynamically calculated deployments. This article walks/runs through creating a new SQL project and deploying it to a database from the command line and any text editor. The tooling ecosystem around SQL projects includes VS Code, Visual Studio, and the SqlPackage CLI. Learn more about all of this at the Microsoft.Build.Sql projects documentation.

⏸️ Before you get started

SQL database projects are built on top of the .NET SDK - you don’t need to know .NET development to use them - but you do need the .NET 8 or higher SDK installed.

Want to quickly check if you have the .NET SDK installed?

dotnet --list-sdks

The SqlPackage CLI is an indespensible part of database deployments because it will automatically calculate the actual scripts necessary to modify a new or existing database to match the contents of your SQL project. SqlPackage is available for Windows, macOS, and Linux all as a .NET tool, so you can install it quickly in a single command:

dotnet tool install -g microsoft.sqlpackage

▶️ With the .NET SDK and the SqlPackage CLI both installed, you’re ready to go.

📁 Create a SQL project

We’ll use the Microsoft.Build.Sql.Templates NuGet package to get started with a new SQL project.

dotnet new install Microsoft.Build.Sql.Templates

Create a new SQL project using the sqlproj template:

dotnet new sqlproj -n ProductsTutorial

📝 Add a database object

Add a new table dbo.Product in a .sql file (dbo.Product.sql) alongside the project file.

CREATE TABLE [dbo].[Product](
    [ProductID] [int] IDENTITY(1,1) NOT NULL PRIMARY KEY,
    [ProductName] [nvarchar](200) NOT NULL
);

🛠️ Build the project

The SQL project (.sqlproj) is the source code for our database definition, and we would use a build artifact to deploy to a database instance (such as a local container or a SQL Server). Build the project to create a .dacpac file.

dotnet build

✚ Add more database objects

Add a stored procedure

We’re going to add a stored procedure to the project. Create a new file dbo.GetProductInfo.sql alongside the project file.

CREATE PROCEDURE [dbo].[GetProductInfo]
    @ProductID INT
AS
BEGIN
    SELECT P.ProductID
        , P.ProductName
        , SUM(PO.Quantity) AS TotalQuantity
    FROM dbo.Product P
    LEFT JOIN dbo.ProductOrder PO ON PO.ProductID = P.ProductID
    WHERE P.ProductID = @ProductID
    GROUP BY P.ProductID, P.ProductName
END

🚧 We’ve added a stored procedure that references the dbo.Product table we created earlier in addition to a table we haven’t created yet (dbo.ProductOrder). Running dotnet build at this point returns build warnings about unresolved references. This is good - we want our development tooling to help us create valid SQL objects.

Adding the missing table

Create a new file dbo.ProductOrder.sql alongside the project file for the missing table dbo.ProductOrder.

CREATE TABLE [dbo].[ProductOrder](
    [ProductOrderID] [int] IDENTITY(1,1) NOT NULL PRIMARY KEY,
    [ProductID] [int] NOT NULL FOREIGN KEY REFERENCES dbo.Product(ProductID),
    [Quantity] [int] NOT NULL
);

Organize the project

We’ve been creating all our files in the same folder with the .sqlproj file. This works, but it’s not the best way to organize a project. A common approach is to group objects by schema and/or object type.

Create a new folder Tables and move the dbo.Product.sql and dbo.ProductOrder.sql files into it. Create a new folder StoredProcedures and move the dbo.GetProductInfo.sql file into it.

When you run dotnet build again, the build still succeeds. The build process automatically finds the .sql files in the project folder and subfolders.

💡 Use your existing databases

If you have an existing database, the command line tool SqlPackage can extract object definitions for the whole database.

sqlpackage /Action:Extract /TargetFile:newfolderforfiles /SourceConnectionString:"Data Source=yourservername;Initial Catalog=yourdatabasename;Authentication=Active Directory Interactive" /p:ExtractTarget=SchemaObjectType

🛳️ Publish a SQL project

Publish a SQL project to a database using the SqlPackage publish command. Learn more about the publish command in the SqlPackage documentation, where additional examples and details on the parameters are available.

# example publish to Azure SQL Database using SQL authentication and a connection string
SqlPackage /Action:Publish /SourceFile:"bin\Debug\ProductsTutorial.dacpac" \
    /TargetConnectionString:"Server=tcp:{yourserver}.database.windows.net,1433;Initial Catalog=ProductsTutorial;User ID=sqladmin;Password={your_password};Encrypt=True;TrustServerCertificate=False;"

When the publish command completes, you can connect to the database and view the objects that were created.

🧩 (optional) Update the project and publish again

We can update the project by adding a new column to the dbo.Product table. Open the existing dbo.Product.sql file and add column for ProductDescription.

The resulting file is:

CREATE TABLE [dbo].[Product](
    [ProductID] [int] IDENTITY(1,1) NOT NULL PRIMARY KEY,
    [ProductName] [nvarchar](200) NOT NULL,
    [ProductDescription] [nvarchar](800) NULL
);

When we run dotnet build again, the build succeeds and the ProductsTutorial.dacpac file is updated.

🪄 We can publish the updated project to the database using the same publish command as before. Running the publish command again will update the existing database schema automatically, there’s no need to remove the database first or track the changes manually.

(optional) Check the project into source control

One of the benefits of using SQL projects is that the database schema is stored in source control. This makes it easy to collaborate and to deploy changes using CI/CD practices.

When we build the project, the .dacpac file and other artifacts are created in the bin and obj folders. We can keep these folders out of source control by adding them to the .gitignore file or by using the default .gitignore for .NET.

The default .gitignore for .NET projects can be created by running the following command in the project folder:

dotnet new gitignore

Once you’ve committed the project to source control, you can push it to a GitHub/remote repository. You can do this from your IDE or from the command line using the following commands:

Commit the project to source control

git init
git add .
git commit -m "new project"

Push the project to a GitHub repository

git remote add origin https://github.com/{yourusername}/{yourrepository}.git
git branch -M main
git push -u origin main

🚀 (optional) Add a GitHub build/deploy pipeline

If we’ve pushed our project to a GitHub repository, we can utilize GitHub Actions to build and deploy our project to a database. This is a great way to automate the deployment of database changes.

The pipeline definitions are stored in a .github/workflows folder. Create a new file build-and-deploy.yml in the .github/workflows folder. We’ll use sql-action to handle building the SQL project and publishing it to a database. It is recommended you check out the sql-action documentation for more information on how to use it, but an example pipeline definition is shown below to get you started.

The example pipeline definition below uses a GitHub secret to store the connection string as SQL_CONNECTION_STRING.

# .github/workflows/build-and-deploy.yml
name: Build and deploy SQL project
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: azure/sql-action@v2.3
      with:        
        connection-string: $
        path: './ProductsTutorial.sqlproj'
        action: 'publish'

📚 Learn more

Wow, you made it to the end! Here are some additional resources to learn more about SQL projects for development and deployment:

Connecting to SQL Server on macOS from Windows VM

2024-01-15T00:00:00+00:00

There are a few instances where you want to run SQL Server on macOS and interact with it from a windows VM. As someone that works within the world of SQL Server/Azure SQL, the use of SQL Server Management Studio (SSMS) comes to mind immediately - for the tasks you can’t complete in Azure Data Studio on macOS directly. Another adjacent (and generally unfortunate scenario) is when work with .NET Framework is required, and for that, you need Windows. This article will step through the network information needed to connect from a Windows VM on macOS to a SQL Server container.

A few basics

I want to rehash a few things quickly as to why we have this specific setup - a macOS host machine running both a container runtime (Docker) and Parallels for VM hosting. The arm64 CPU architecture of modern macOS machines as part of their SoC (system on a chip) hardware is just about magical (not a technical term) compared to a lot of other contemporary machines, especially factoring in the power sipping consumption, but a number of components are not natively available. SQL Server, for example, is only available as an x64 installer for Windows/Linux and an x64 container.

Operating systems are able to offer integrated emulation layers, such that they can run applications that don’t match their CPU architecture. For example, Windows on arm64 can run the x64 SQL Server Management Studio with a slight performance hit due to the emulation layer. On macOS, the emulation layer is able to interact with the Docker container runtime and run containers built for x64.

Your best bet for running a Windows VM on macOS is to run an arm64 image of Windows in Parallels, and within the VM you can run the Windows applications you need - except SQL Server itself.

Creating a SQL Server container on macOS

VM networking

Shared networking is the recommended (and default) network setting in Parallels. In this network layout, your macOS machine is automatically sharing its network connection with the VM and this is usually sufficient for VM use. The other insight we need to connect between macOS and the Windows VM is knowledge of the virtual subnet (network) between the macOS host and the VM.

Confirming the VM has shared network set

The fastest way I’ve found to get my network info for the VM is within the VM itself, using ipconfig from Windows terminal. From its output, I note the VM’s IP address and its gateway address. On my machine, the VM has the address 10.211.55.3 and the gateway is 10.211.55.1.

ipconfig showing my VM IP address is 10.211.55.3

Conceptually, our macOS machine is playing 2 roles in the VM network - both the gateway and as a participant in the network. The SQL Server container running on our macOS host is available to the Windows VM at the IP address assigned to our host as a participant in the network.

The IP address in question

Since the gateway address is x.x.x.1 and the VM is assigned x.x.x.3, I’m going to made an educated guess that my host machine is available at x.x.x.2. All that IP address information I gathered by running ipconfig from the Windows VM.

Test the connection

Using any SQL client application on the Windows VM will allow me to test the connectivity to the SQL container. This moment is a good one to fire up SSMS in the VM. I’ll use the address 10.211.55.2 as my SQL server address.

Connecting in SSMS

Good news everyone

If you encounter difficulties gaining connectivity, you may need to adjust firewall settings in macOS. If your firewall is enabled, use the Options button to allow specific connections.

Recap

While there’s a lot of the development process that you can accomplish from just about any machine, there are still some specific instances where you need that Windows VM. With the default shared network setting, your macOS host has an address assigned to it that you can determine and use to connect from the Windows VM to a container running on macOS. In the setup on my machine, the Windows VM was assigned 10.211.55.3 and the macOS host (and SQL Server container) was available at 10.211.55.2.

Using Python and Azure Functions to send data from Azure SQL Database

2023-02-26T00:00:00+00:00

When building applications on Azure SQL, one of the most flexible ways to send data from your database to other systems is to use Azure Functions. Azure Functions are serverless functions that can be triggered by a variety of events, including HTTP requests, timers, and Azure SQL Database changes. In this article, we will discuss how to send data from an Azure SQL Database to an FTP server and API endpoints using Azure Functions. The complete sample code for this article is available on GitHub.

This post is syndicated from https://devblogs.microsoft.com/azure-sql/using-python-and-azure-functions-to-send-data-from-azure-sql-database/

Get data from Azure SQL Database in Azure Functions

With Azure SQL bindings for Azure Functions we can easily retrieve data from an Azure SQL Database in an Azure Function, leaving the boilerplate code of connecting to the database and executing queries to the Azure Functions runtime. When our solution needs to operate on a schedule, such as every morning, we can use the timer trigger to start the Azure Function. Python Azure Functions are composed of a function.json file and an __init__.py file. The function.json file is where we define the function trigger and input/output bindings and the Python code is based in the __init__.py file. Querying Azure SQL Database with an Azure Function is as simple as adding an input binding to the function.json file:

{
    "name": "products",
    "type": "sql",
    "direction": "in",
    "commandText": "SELECT [ProductID],[Name],[ProductModel],[Description] FROM [SalesLT].[vProductAndDescription]",
    "commandType": "Text",
    "connectionStringSetting": "SqlConnectionString"
}

Once we have the input binding defined, we can use the parameter products in our function code to access the data returned by the query. The products parameter is a list of SqlRow objects, which are similar to Python dictionaries.

The Azure SQL input bindings for Azure Functions can run any SQL query, including stored procedures. The commandText property is where we define the SQL query to run. In the example above, we’re selecting four columns from the view SalesLT.vProductAndDescription. The connectionStringSetting property is where we define the name of the app setting that contains the connection string to the Azure SQL Database. Additional examples are available which show using additional features, including parameters and executing SQL stored procedures.

Throughout the sample we have several values in the Azure Functions application settings, including the Azure SQL connection string, the API endpoint URL, and the FTP server login information. Keeping this sort of sensitive information out of code is a best practice that you’ll want to follow.

Sending data to an API endpoint

To send data to an API endpoint, we will use the requests library for it’s simplicity and the built-in json library. With the requests library, we can easily send a POST request to an API endpoint with the data we want to send. The SQL input binding sends data as a list of SqlRow objects, which are similar to Python dictionaries. We can use the json library to serialize the data into a JSON string, which is the format that most APIs expect.

def main(everyDayAt5AM: func.TimerRequest, products: func.SqlRowList) -> None:
    logging.info('Python timer trigger function started')
    # convert the SQL data to JSON in memory
    rows = list(map(lambda r: json.loads(r.to_json()), products))

    # get the API endpoint from app settings
    api_url = os.environ['API_URL']

    # send the data to the API
    response = requests.post(api_url, json=rows)
    # check for 2xx status code
    if response.status_code // 100 != 2:
        logging.error(f"API response: {response.status_code} {response.reason}")
    else:
        logging.info(f"API response: {response.status_code} {response.reason}")

In our Azure Function we check the API response status code to make sure the request was successful. If the status code is not in the 2xx range, we log an error. If the status code is in the 2xx range, we log a success message. By logging an error, we can monitor the Azure Functions logs to see if there are any issues with calling the API endpoint.

That’s it! Those ~10 lines of Python are all we need to run a query against our Azure SQL Database and send that data to the endpoint we set in the Azure Functions app settings.

Sending data to an FTP server

While we formatted the data as JSON to send to an API endpoint, we may want to send our data to an FTP server as a CSV file. By using a package like pandas, we can quickly convert the data to a comma-separated format.

def main(everyDayAt5AM: func.TimerRequest, products: func.SqlRowList) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')
    filename = "products.txt"
    filesize = 0

    # convert the SQL data to comma separated text
    product_list = pandas.DataFrame(products)
    product_csv = product_list.to_csv(index=False)

Python has a built-in library, ftplib, that can interact with FTP servers. After retrieving the FTP server information from the app settings, we can connect to the FTP server and upload the data. Instead of writing the data to a local file before uploading to the FTP server, we can use the BytesIO class to handle the binary data to memory.

    datatosend = io.BytesIO(product_csv.encode('utf-8'))

    # get FTP connection details from app settings
    FTP_HOST = os.environ['FTP_HOST']
    FTP_USER = os.environ['FTP_USER']
    FTP_PASS = os.environ['FTP_PASS']

    # connect to the FTP server
    try:
        with ftplib.FTP(FTP_HOST, FTP_USER, FTP_PASS, encoding="utf-8") as ftp:
            logging.info(ftp.getwelcome())
            # use FTP's STOR command to upload the data
            ftp.storbinary(f"STOR {filename}", datatosend)
            filesize = ftp.size(filename)
            ftp.quit()
    except Exception as e:
        logging.error(e)

    logging.info(f"File {filename} uploaded to FTP server. Size: {filesize} bytes")

Wrapping up

With Azure Functions we have a low-overhead and flexible way to build application components and the Azure SQL bindings make it easy to retrieve data from Azure SQL Database. In this article, we took a brief look at an approach to sending data from an Azure SQL Database to an FTP server and API endpoints with Python in Azure Functions. If you’d like to dive into this sample further, the code is available on GitHub.

OMSCS Research Options

2023-01-03T00:00:00+00:00

The Georgia Tech online masters of science in computer science has been a groundbreaking endeavour to bring an MS CS program to students at a massive scale.

In five years, the program has received over 25,000 applications and enrolled more than 10,000 students (including those who have graduated), all working their way toward the same Georgia Tech M.S. in Computer Science as their on-campus counterparts. - https://omscs.gatech.edu/explore-oms-cs

Despite the use of online communities, legions of teaching assistants, and other techniques to expand the reach of a traditional masters of science program to thousands of students, there are still opportunities for students to be deeply engaged with university resources. One of these categories of opportunities is research and independent projects, which I was fortunate to experience in three ways during my time in the program.

Education Technology Foundations (CS6460)

Previous review comments here.

I’m starting with the EdTech course because the barrier to entry is low and the pathway to success is well guided. Registration for EdTech (CS6460) is open to OMSCS students without additional forms and even if you don’t have a project in mind before the course, the first few weeks of the course involve exploration of academic literature on the intersection of education and technology. If you do have a project in mind, the first part of the course will help you focus and refine your idea to best fit the remainder of the course.

To be specific about your success in this course, you will agree on an outcome with a TA/mentor along with a planned approach when you decide on your topic. Along the way, adjustments can be made so its in your best interest to be honest with your mentor about challenges you are having or unforeseen barriers. In many cases your mentor will have experience in or adjacent to the area you’re working on and can provide helpful suggestions. Much of this is similar to working with a PI (principal investigator) in a research group except the EdTech course takes a more structured approach to the project plan than most research groups.

After EdTech, you can optionally take your project further through a few avenues:

participate in the student showcase, which is an online forum format of student presentations at the end of each semester
continue working on the project independently, which is especially applicable if you are building an open source component
approach a professor working in a related area and inquire if they’re willing to mentor you for continued work through CS8903 (see below), a number of EdTech projects will fall in Dr. Joyner’s areas of interest

In my opinion, unless you do 2-3x more work during the EdTech course than is required for the course, you are unlikely to have a project ready for publication in academic journals at the end of this class. However, if you continue the project past the course you may be on the way to publication. Regardless, EdTech can be a tremendously rewarding adventure into working on an individual or team project focused on an area of your interest.

Vertically Integrated Projects (VIP)

Previous review comments here.

OMSCS students are eligible to apply for VIP teams, an email is usually sent once a semester to remind students but you can begin investigating options at any time. Applications are open around the time of phase 1 registration, so you need to be thinking ahead if at all possible.

My suggestion for participating in VIP is to not only complete the application to a team, but if you have related experience (professional or academic) or compelling interest in specific topic that you also communicate with the professor leading the team via email. Do enough background research into the work the team is doing to have engaging questions.

An important component of VIP is the team component. There is a high likelihood that you will be collaborating with other students, undergraduate and/or graduate students. This makes VIP an excellent opportunity to learn from others but also to be a leader and help other students. OMSCS students can be highly desireable to VIP teams because we come to Georgia Tech with diverse backgrounds and experiences, exactly the kind of innovation that vertically integrated projects are seeking.

My experience with VIP involved relating previous undergraduate and graduate research to the VIP project components, as well as areas I’m interested in learning more about. The VIP team I joined had a branch that required some development experience that I was able to work on where my previous experience was helpful in gaining context quickly and getting started.

Special Projects (CS8903)

Previous review comments here.

Special projects does not change your OMSCS degree pathway from being coursework based, but it does create a significant opportunity to do research for credit (3 or 9 credits) that can be part of your coursework credits. These are credits that count towards your “free electives”, no matter what your specialization is. These statements are made based on the current content of https://omscs.gatech.edu/program-info/specializations, you will check with your academic advisor before pursuing.

Understanding academic research

Academic faculty are highly motivated to conduct research, where there success is measured mostly by their publications but also by the success of their team (postdocs, graduate students, and extraordinary undergrads). The phrase “publish or perish” is invoked as a reminder that research must go on. This is not to suggest that all academic faculty aren’t interested in your success as a student, but it’s really important to understand some of these dynamics if you want to successfully participate in academic research. First off - I’ll define your success as an OMSCS student in CS8903:

Making forward and novel progress on a project of sufficient difficulty that it makes for engaging conversation with people on a deep technical level
Completing agreed upon checkpoints to complete the CS8903 credit hours. If you were a full time research student, you would be trying to meet expectations to continue your “employment” in the group
Optionally, your work is published or included in a publication by the research group

Research groups are generally structured horizontally into subgroups of people around projects under the PI (principal investigator, or faculty member). A research group around a PI can contain post-doctoral researchers (postdocs), graduate/PhD students, and undergraduate students. In larger groups, the more senior folks (postdocs) can take on a good deal of mentorship of student researchers. Newer faculty members will have smaller groups that are often more tight-knit in both the vertical and horizontal direction.

Getting Started with CS8903

The process for starting CS8903 is a bit more involved than the previous options, but the possibilities are nearly endless! There are over 100 academic faculty in GT Computing and all of their research areas are potential content for your work for CS8903. To enroll in CS8903 and work on a faculty member’s research project, you will need to:

Educate yourself on their current work, including publications and opportunities for more advancement in that area. You may want to familiarize yourself with the GT Library to get access to some of the content.
Reach out (via email) to introduce yourself and/or complete any research interest forms they have linked on their website (not all faculty have these). Similarly to the introduction email I recommend for VIP (although now required), you’re trying to make a good impression - you’re smart, inquisitive, hard working, etc.
Ask the professor if they’re willing to supervise you for CS8903 where you’d work on their project {fill in the blank here} based on your interest/experience/knowledge/unique charm. It’s ideal to have a virtual meeting with the professor mid-semester to discuss their work and brainstorm what you might work on. Look at the CS8903 permit form before this meeting.
If the professor agrees, complete and submit the CS8903 permit form, including a statement of research.
Register for CS8903 during registration and setup regularly meetings with your advisor. If their research group does virtual or hybrid meetings, you may want to join those.

The statement of research should be two or three pages and should include the following:

Problem Statement or Project Goals

Solution Proposal or Approach

Schedule of Work

Expected Results or Outcome (Deliverables)

School of Computer Science Special Problems (CS 8903) Permit Form, https://www.cc.gatech.edu/graduate-forms-procedures

I got started with CS8903 by digging further into the “Databases” area of research and looking through all the faculty at Georgia Tech I could find that work with databases. Prof. Arulraj is involved in a few projects, some newer and others carrying over from their original PhD work. Their research group is relatively small - which is a pro or con depending on a number of other factors. One of those projects was of significant interest to me, so I read no fewer than 10 papers of theirs or referenced by them before reaching out to ask about future opportunities.

Summary

This article isn’t necessarily exhaustive and there may be other ways to conduct research in OMSCS, including reaching out to a professor similarly to as would be done for CS8903 but instead of under the premise of you doing research work “for free”. As the OMSCS program evolves over time, additional opportunities may become available for students. In my experience, all three of VIP, EdTech, and CS8903 options were readily available to customize the work done for OMSCS to fit my area of interest. Know your strengths and don’t be afraid to explore!

OMSCS Graduation and Recap

2022-12-31T00:00:00+00:00

The online Masters of Computer Science program at Georgia Tech is completely online and designed for working, part-time students. I applied to OMSCS in 2015 when I was concerned about what I saw as a plateau ahead in my career. At the time I worked for a smaller company doing a little bit of everything in technology and development. With a MS in chemistry and a minor in computer science under my belt I was confident that I would be able to succeed in OMSCS and it would contribute to my part of my aspiration to be a better developer and work as software engineer in the future.

I received the decision notification email while driving from Missouri back home to Minnesota. It was unpleasantly cold and windy at a gas station in Iowa as I navigated the applicant portal from my iPhone. Also relatively unpleasant was the news that I had not been accepted to OMSCS.

In December 2022 (this month), I graduated from OMSCS. In this post I’m going to talk just a bit about what I did between being rejected in 2015 and being admitted to OMSCS as well as review some pieces of OMSCS.

Pre-OMSCS

I’m not going to speculate about the differences in my applications and the applicant pools between 2015 and 2020, but I do want to talk about how I was a different person going into OMSCS because of the additional experiences I did have in that time.

I developed a few skills for Amazon’s Echo/Alexa devices, and as a part of it started thinking more developer experiences and platform integration. I even took a day off once to go to an Alexa Dev event in Minneapolis “for fun”.
I got involved in professional organizations around the systems I used at work (SQL Server, Dynamics SL). With SQL Server I was able to get back to speaking/teaching and learned a whole lot from the great folks involved with SQL Saturday MN (PASSMN). With Dynamics SL I was able to learn more about the depths of the product in organizing content for conferences and eventually served on the Board of Directors.
I took a few certification exams to solidify and validate technical knowledge.
I completed two professional certificate programs through Coursera focused on product management and business strategy.
I created a few extensions for Azure Data Studio.

To be honest, I wasn’t planning to re-apply for OMSCS. It was in early 2020 when I was looking to apply for new roles (landing in SQL experiences at Microsoft) that I decided to go after OMSCS again. I figured if one avenue didn’t work out, maybe the other would.

OMSCS Review

I worked on the “Computing Systems” specialization and over 2.5 years completed the requirements for matriculation (2020-2022). COVID-19 restrictions were in full force during the early semesters - which had its benefits at times - but we had just moved to Washington and I had just started a new job. In later semesters all 3 of our dogs passed away. There are some colleges that have hybrid or online MS CS programs that still have significant synchronous or on-campus components, but Georgia Tech’s OMSCS is fully-centered in the online campus.

Despite all the daily stresses and life events happening during OMSCS, my key to success was repeatedly setting aside everything else to focus on course content for hours every week. For some classes this might be in the range of 10 hours, for others it was 20+ hours. Given the demands of life, I am incredibly fortunate to have a supportive spouse who made a number of sacrifices over that span.

There are some realities of a graduate program that can be jarring to the student, especially if they’re expecting more of a bachelors degree:

the pace at which material is covered requires individual learning - know how you learn topics
the topics/subjects are provided, but the materials that you need to master it may not be explicitly provided - know how to look for books, papers, videos
the background of each student varies widely and some folks grasp topics faster than others - know how to leverage study groups or ask peers for advice

Courses

Fall 2020

The common advice for the first semester is to take only a single class to allow you to adjust to graduate school and avoid becoming overwhelmed. I didn’t listen and added the VIP project course, but I also didn’t really read course reviews yet either.

Vertically Integrated Project (Big Data and Quantum Mechanics):

⭐️⭐️⭐️⭐️_ (4/5 stars)

The Vertically Integrated Projects (VIP) program is a cross-functional research initiative at Georgia Tech where bachelors and masters students can contribute to larger projects alongside faculty and PhD students, where the increased innovation benefits the multidisciplinary projects and students earn course credit for their work. As a student, team assignment is a selective process, but for OMSCS students there are a good number of teams that are seeking CS expertise for their projects. I applied to the the Medford group to work on their visualization software for ML-assisted DFT (ElectroLens). The team was an ideal fit due to my original MS in chemistry where my thesis leveraged DFT and ElectroLens is based in Electron/JavaScript. I can’t recommend VIP highly enough, whether you have previous research experience or are looking for more guidance - you get great exposure to academic research through this program.

Database Systems Concepts and Design (CS6400):

⭐️____ (1/5 stars)

This course moved fairly slowly through entity-relationship diagrams, relational algebra/calculus, and report writing for a LAMP stack application. The preface for the textbook used literally recommends the chapters covered by this class for an introductory undergraduate course. I was sorely disappointed at the lack of coverage for index structures, query processing, transactions, or security (all included in the textbook for use at the graduate level). Frankly, I was so unimpressed with the databases course that if I hadn’t also done VIP that first semester I would have been tempted to stop the program and focus on promotions at work. I was concerned that every course would have such low quality of content. I didn’t find this to be true, and am glad I continued on to better courses.

group project (you can get team members who do nothing)
rudimentary content all the way through
terribly worded exam questions

Spring 2021

After carefully reading course reviews, I paired an easier course (Computer Networks) with a more time-consuming course (Intro to OS), both well-rated.

Computer Networks (CS6250):

⭐️⭐️⭐️⭐️_ (4/5 stars)

My systems administration background served me well in this class, which had a nice balance between exams/quizzes/projects. The content looked at how some networking fundamentals work behind the scenes and the projects were often Python implementations of algorithms. Extra kudos to the instructional design for this class such that students could collaborate on creating test cases and/or unit test frameworks for the projects, which created additional learning opportunities.

easy-to-moderate assignments
content chopped up into bite-size segments

Graduate Intro to OS (CS6200):

⭐️⭐️⭐️⭐️⭐️ (5/5 stars)

This course is legendary for its content, project time commitment, and instruction. The depth and pacing of the topics is not overwhelming, but studying for the exams with flash cards was necessary to make sure I was picking up the material sufficiently. On top of preparing for exams with the course videos and reading, this course’s projects (3 of them) are each about a 40 hour commitment (varying based on your experience with C, debugging, and acclimating to an existing codebase). I had at least 1 near-breakdown with each project, often at late hours of the night, as my brain stretched to grasp problem space and properly allocate memory. If I had to pick a favorite class from the program - this would have been it. After completing the course I wanted to take advanced operating systems but ultimately didn’t fit it in.

2 moderate difficulty exams that aren’t too highly weighted
3 difficult assignments with clear connection to course material

Summer 2021

Some students opt to have the summer free from school and don’t take a class, especially since it’s an abbreviated semester. I opted for enrolling in classes that didn’t have enormous workloads during the summers.

Software Arch & Design (CS6310):

⭐️⭐️___ (2/5 stars)

While this course was full of busy work through in-module quizzes, it also offered a lot of supplementary/required readings. The most notorious is likely the Gang of Four design patterns, but there many others including some solid nods to Uncle Bob. The course had about as many assignments on how to communicate the software architecture through UML/OCL as designing the architecture, which is valuable but there’s a depth of architecture design content available that is interesting to check out. The bulk of the grade for this course came from project assignments, which culminated in a group project component. Similarly to the first semester group project, there was a group member who ultimately shouldn’t have received any points because despite multiple people trying to bring them up to speed they were unable to build the project code much less contribute. As long as your group only has 1 or 2 people in this category (out of 4 or 5), you’ll be able to pull through and complete the work, but it is a risk. I’m a huge supporter of the program’s high acceptance rate but I propose that it is reasonable that a subset of courses are approved for first-semester students that have only individual work to bottleneck students who need to ramp up.

most valuable course content is in supplemental material
group project
“easy course” - highest grade in the program

Fall 2021

Individually both of these courses are notable for content and instructional style, but they additionally share a significant amount of academic paper reading and writing. To spare myself the context switching, I paired the courses up for a semester where I could stay in the mindset of interaction design and writing.

Human-Computer Interaction (CS6750):

⭐️⭐️⭐️⭐️⭐️ (5/5 stars)

HCI is a “Dr. Joyner course” with 2 aligning sets of papers focused on the principles learned and applying those as methods towards a project idea. There’s a good amount of engaging reading, and it certainly helps that I’m interested in the topic. I still some of those resources from time to time for work as a PM. As tempting as it was to use one of the SQL tools for the project, I ended up proposing improvement to the GPS navigation component of Apple CarPlay, and had a little fun along the way with user surveys and scraping product review APIs.

Education Technology Foundations (CS6460):

⭐️⭐️⭐️⭐️_ (4/5 stars)

EdTech is another “Dr. Joyner course” and is similarly structured to HCI but with a much larger independent project component. You are given a general direction to investigate a gap in education where technology plays a role, and can conduct empirical research or implement improvements. My focus area, to no one’s surprise, was database systems. After scrubbing the literature on higher education for database systems, one of the chasms that forms is automated grading infrastructure for database assignments. I took a swing at the issue with SQLGrader, designed to scale to courses such as the databases course I took in the first semester.

SQLGrader architecture overview

I can’t say I continued working on SQLGrader despite there being many opportunities to improve it, as I had also started a project proposal for the next semester.

Spring 2022

Special Problems (CS8903):

I was very fortunate that Prof. Joy Arulraj was receptive to my inquiry and project proposal, which took tiny step forward with his SQLCheck tool for identifying antipatterns in SQL code. The second version of it was fresh off of feature in VLDB. The work I did was spit into 2 categories - the first was minor/quick improvements to the original (open source) SQLCheck and the second was to establish forward motion on the second generation of SQLCheck. This involved updating dependencies, hardening the container images with a few best practices, and establishing patterns for sharing compute resources with other projects in the DB research group (egress networks, SSL configuration, etc). The Special Problems course is designed to mix a thesis-driven Masters degree with a coursework-driven Masters degree such that the learner can dip their toes in or immerse further in a research area. I won’t give this a star rating because it is a phenomenal option for OMSCS students and Dr. Arulraj was a fantastic mentor, but I know my mental state was rapidly declining during 2022 and had I been engaged on this at a different time would have wished I got more out of it.

I was very worn out from the program at this point and on top of it 2 of our dogs passed away during the spring semester. I’m very glad that I stepped back to taking 1 class at a time to avert total meltdown and to afford more time to spend with Ellie and Louie during their final days.

Summer 2022

Software Analysis and Testing (CS6340):

⭐️⭐️⭐️__ (3/5 stars)

The Software Analysis course covered automated testing/analysis concepts, for example looking at how compilers can detect issues like uninitialized variables in code. Many of the topics were applicable to the security of software (eg fuzz testing) but there’s a strong link to the importance of the tools that are used as software is written (or compiled). A good portion of the assignments/labs were leveraging LLVM C++ APIs, however an attempt was made to use TypeScript/JavaScript in a lab to demonstrate type-checking and delta debugging was explored in Java. No complaints about the variety of languages, only complaint was that the course seemed like it attempted to hard to be “light” with short assignments and only a single exam. It’s not an exaggeration that a handful of the assignments were less than 60 minutes of work. This course could be significantly improved with more investment into the course assessments. Ok course and on the edge of being a good course.

easy-to-moderate assignments, most of the difficulties coming from lab requiring an older version of LLVM
content chopped up into bite-size segments

Machine Learning and Data Science Tooling Seminar (CS8001):

The seminar concept was introduced just in 2022 and I was thrilled! These offerings are 1-credit hour options to interact with others and work through a breadth of light material on a topic, ideal for semesters where you have just a little extra time expected but not enough for a full course. I very much recommend trying out even just 1 during OMSCS.

Fall 2022

Introduction to Graduate Algorithms (CS6515):

⭐️⭐️⭐️⭐️_ (4/5 stars)

I’m torn on the rating for Intro to Graduate Algorithms because it is a good course (4 stars) but it is a graduation requirement and structured differently than most other courses in the program, leading to unnecessary stress for the ~850 students each semester as they scratch and claw their way out (2 stars). The downfall of Georgia Tech’s popularity is that this course bottlenecks at the end of the program and is all but impossible to register for until your last 1, maybe 2 semesters. A number of students take it more than 1 time to receive a passing grade and I understand that there’s no real incentive to have students take this course earlier when they might quit the program instead of toughing it out in this course. It very much reminded me of teaching science for ITT Technical Institute to first-semester students and how the dean of students would come remind me, frequently, that I was grading too harshly and reducing the enrollment numbers for later courses. Many other courses that were difficult to enroll in now have a reduced bottleneck so I hold out hope that in a few more years, graduate algorithms won’t be the last class every student takes.

This course is notoriously difficult, almost a self-perpetuating cycle at this point, since a good portion of the difficulty comes from our responses to stress and inability to think under pressure. If you, like many other students in the program, are far removed from your last formal discussion of algorithms, learning how to even answer homework and exam questions will be difficult. That’s ok, you can handle that. You’re going to need to do a lot of practice problems anyway (dynamic programming, graph algorithms, divide and conquer, linear programming) to really grasp the material, and you’re going to do them until you don’t need your notes or any prompts to work out the answer completely. This class isn’t a leetcode prep course, and although it does introduce some concepts that can be helpful in that prep you will write < 200 lines of code for the class. Office hours (mini lectures) are provided weekly, which you can go to live or watch the recording. Study groups are recommended, and hopefully you are unafraid to leave/join study groups until you find one that works well for you. In your study group you should be able to ask questions and work through problems while your peers provide feedback (criticism, reinforcement, etc). When all is said and done, this class has three exams that are each 25% of your grade. The final exam is only an opportunity to replace one of those exam scores if you’d like to improve the grade you’re at, an opportunity I’ve heard isn’t available during the summer. I was sitting in the B range before the final exam and was quite happy to not need to take it.

Research

If you made it through my course reviews, you probably noted that I had a few opportunities to do research.

VIP
Education Technology
Special Problems

Despite being a fully-online in both coursework and community program, Georgia Tech manages to offer multiple avenues for OMSCS students to do research - which is phenomenal. Whatever your aspirations are following OMSCS, there’s an opportunity to dive deeper into a subject that interests you if you’re ready to work independently.

Cost

Fall 2020: $1,381
Spring 2021: $1,381
Summer 2021: $841
Fall 2021: $1,381
Spring 2022: $841
Summer 2022: $1,021 (increased due to seminar enrollment)
Fall 2022: $647 (decreased for all OMSCS students with removal of institutional fee)

Total: $7,493

(One of my employer’s benefits is a generous amount of tuition reimbursement, which I used to cover the cost of OMSCS.)

Summary

I’m proud of having completed OMSCS and I’m really grateful for all the learning opportunities I had as a part of the process. I don’t have the same aspirations as I did during my first application to the program but I do know that even while in the program the knowledge I was picking up was immediately applicable to my work, to which I’ll credit some of my growth at work to OMSCS. It was a tough program and offers a lot of options for students to create their path that benefits them the most.

Sometimes people would ask me “what’s it like working and doing school part time?” or “I’ve been thinking about doing a masters, how is it?” - to which my response is usually:

It’s really awful and painful, but I love it.

Thanks OMSCS, I loved it.

Azure Logic Apps and Form-Data HTTP Requests

2019-03-06T00:00:00+00:00

The nuts and bolts of this post is about sending an HTTP POST request in an Azure Logic App that utilizes the multipart/form-data content type. I don’t run into it often, but when I do, I’m sure glad I figured out how to do more than application/json request bodies in Logic Apps. The use case I came across this week for multipart/form-data body was for the Mailgun API.

The Mailgun example

curl -s --user 'api:YOUR_API_KEY' \
    https://api.mailgun.net/v3/YOUR_DOMAIN_NAME/messages \
    -F from='Excited User <mailgun@YOUR_DOMAIN_NAME>' \
    -F to=YOU@YOUR_DOMAIN_NAME \
    -F to=bar@example.com \
    -F subject='Hello' \
    -F text='Testing some Mailgun awesomeness!'

The above example is directly from the Mailgun documentation, but we need to translate it into an Azure Logic APPS HTTP request.

This is a POST request
The URL is https://api.mailgun.net/v3/YOUR_DOMAIN_NAME/messages
You need a header for “Content-Type” with a value that sets form-data and a boundary. This example value would work just fine “multipart/form-data; boundary=—-WebKitFormBoundary7MA4YWxkTrZu0gW”
We’ll cover the body text area in a moment.
Authentication is basic, username is api and the password is your private api key from the mailgun dashboard.

This is a POST request - The URL is https://api.mailgun.net/v3/YOUR_DOMAIN_NAME/messages - You need a header for “Content-Type” with a value that sets form-data and a boundary. This example value would work just fine “multipart/form-data; boundary=—-WebKitFormBoundary7MA4YWxkTrZu0gW” - We’ll cover the body text area in a moment. - Authentication is basic, username is api and the password is your private api key from the mailgun dashboard.

Request Body

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="from"

Excited User &lt;mailgun@YOUR_DOMAIN_NAME>
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="to"

YOU@YOUR_DOMAIN_NAME
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="subject"

Hello
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="text"

Testing some Mailgun awesomeness!
------WebKitFormBoundary7MA4YWxkTrZu0gW--

The body content is comprised from sets of 4 rows following the form field boundary value:

label for the form field, such as Content-Disposition: form-data; name=”from”
empty line
form field value
the form field boundary value as stated in the Content-Type header value

Through Postman

If you’re working with HTTP requests I highly recommend a tool such as Postman to test, save, and modify your endpoints. In the instance of creating a form-data request for Azure Logic apps, the “Code” functionality in Postman can save you a bit of time.

You would start by building the POST request you would like to make in Postman, including the form values in the Body tab.

After the information is entered into Postman, click the small “Code” action in the upper right, which opens a popup with the request build in a selection of languages. We want to switch HTTP to Java OK HTTP.

Two sections of code are important to grab from here for use in Azure Logic apps. The first is the highlighted request body code, highlighted. The second is the header value for content-type on line 8, which is "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" in this example.

Back in the Azure Logic apps UI, switch from Designer to Code view. You will paste the request body in for the body value here.

You will need to complete the rest of the request values, including the Content-Type header, for your request to be complete.

Azure Logic App HTTP Requests

I love leveraging Logic Apps to simplify the interaction from the endpoint application and reduce potential change points when a 3rd party has API changes. Most frequently I’m hitting a POST endpoint that accepts a JSON body, but I still occassionally see form-data. By creating a boundary string in the header and setting the body up with the 4-line sets (label, blank, value, boundary), you can send these requests in Azure Logic Apps.