Will ChatGPT replace PowerShell developers?

How’s that for a click-bait title? Seriously, I’m not trolling for clicks.

But recently I learned on a “Security Now!” podcast that OpenAI’s ChatGPT had created the basis of a PowerShell script that helped assess the state of a LastPass user’s vault. (LastPass had recently admitted to losing customers’ vaults in a hack.)

This discussion intrigued me. I’ve been working in and enjoying the flexibility of PowerShell for about a decade. Now cross-platform, PowerShell has made my life easy as both my default shell and an exceptionally powerful scripting language, largely because both Azure and AWS fully support the language and its .Net objects.

So, I decided to try ChatGPT for myself. I asked it to write a script that would “convert nested JSON objects”. You can see the query I gave the AI in the screenshot below.

As you can see, I was a bit vague. I didn’t specify what I wanted the output to be. And I didn’t bother to provide any input as an example of what to “convert.” Still, what the AI produced had me very excited — at first. Here’s the original PowerShell script ChatGPT produced.

# Define the input JSON object
$json = '{
    "level1": {
        "level2a": {
            "key1": "value1",
            "key2": "value2"
        },
        "level2b": {
            "key3": "value3",
            "key4": "value4"
        }
    }
}'
# Convert the JSON object to a PowerShell object
$obj = ConvertFrom-Json $json
# Create an empty object to store the flat key-value pairs
$flatObj = @{}
# Function to flatten the object recursively
function Flatten-Object {
    param(
        [hashtable]$object,
        [string]$prefix
    )
    foreach($key in $object.Keys) {
        $value = $object.$key
        if($value -is [hashtable]) {
            Flatten-Object $value "$prefix$key."
        } else {
            $flatObj.Add("$prefix$key", $value)
        }
    }
}
# Call the function to flatten the object
Flatten-Object $obj ""
# Output the flat key-value pairs
$flatObj

I was impressed by this output. It’s sparse, well-commented and uses a recursive function to accomplish the task. It also provided its own input data in the $json variable. I was expecting much less. I assumed from this cosmetic review it would just run and produce a listing of the JSON structure it defined on the console.

Not so fast…while the code is syntactically correct, it’s non-functional. At least in this way, ChatGPT is a pretty good simulacrum of a human. You’re probably able to count on the fingers of one hand the number of times something you’ve written runs as expected on the first try.

Do you see it? It’s an object type mismatch. ChatGPT’s code created an object (in variable $obj) of type PSCustomObject but the Flatten-Object function expects an object of type Hashtable.

IOW, ChatGPT didn’t understand the type of data objects it was dealing with. That’s a really interesting result, no? This is an area of significant complexity in PowerShell (and, I presume other object-oriented languages). And while sometimes the PowerShell interpreter can help you out implicitly when you need it this isn’t one of those cases. In short, for all its elegance in producing a well-structured script, ChatGPT doesn’t really know much about PowerShell.

Whew! That’s a relief, eh?

Here’s my corrected version of the script and, below it, a screenshot of the output. All it took to correct the generated script was to change its object handling to deal directly with PSCustomObjects. The resulting script still retains ChatGPT’s clever use of a recursive function and its sparse style.

After this, I am left wondering if maybe the future is a bit like the past, only this time applied to code: machines will do some initial heavy lifting coding that humans used to do (as here, proposing a framework in which to solve a problem) and humans will take that input as an accelerant, using our still-better-than machines experience and knowledge to finish the raw input from the AI.

In any case I and my fellow PowerShell devs don’t need to lose any sleep at night — for a while, at least.

# Define the input JSON object
$json = '{
    "level1": {
        "level2a": {
            "key1": "value1",
            "key2": "value2"
        },
        "level2b": {
            "key3": "value3",
            "key4": "value4"
        }
    }
}'
# Convert the JSON object to a PowerShell object
$obj = ConvertFrom-Json $json
# Create an empty object to store the flat key-value pair
$flatObj = @{}
# Function to flatten the object recursively
function Flatten-Object
{
    param(
        $object,
        [string]$prefix
    )
    foreach ($item in $object)
    {
        if ($item -is [PSCustomObject])
        {
            $item.psobject.Properties | ForEach-Object { 
                $name = $_.Name 
                $value = $_.Value
                Flatten-Object $value $name
            }           
        }
        else
        {
            $flatObj.Add($_.Name, $_.Value)
        }
    }
}
# Call the function to flatten the object
Flatten-Object $obj ""
# Output the flat key-value pairs
$flatObj