Category: Science and Philosophy

I’ve been working on a programming language project for quite a while. Its been challenging and fun as my on-again-off-again side project for probably 10+ years.

Needless to say this programming language isn’t known by many people and it is fairly original while borrowing from a few other similarly obscure language projects.

The language is a parser generator language, a programming language programming language if you will. Its intent is to make it easier for people to create DSLs and the accompanying tools. You can check it out at https://github.com/justinmchase/uffda

So I figured it would be an interesting opportunity to see how good the new Big AI actually is by asking it to do something cognitively difficult for a human and which there is little to no literature available online for it to have been heavily trained in prior to my request.

So I asked it to “write a program in the Uffda language that parses Morse Code?”

It gave me a little guff saying that Uffda is actually a scandanavian expression, which is closer to what I expected it to do. But then it went on to say that its actually a TypeScript library for a parser generator and offered “If you want, I can write a program in TypeScript that uses the Uffda library to define a parser for Morse Code.😊”

It then gave me the option to press a button to “Yes, please write the program.”

And so I did… And the results kind of astonished me. It actually wrote what appears to be a viable parser and beyond that it took advantage of a few concepts which really suprised me:

• The TypeScript code was correct (unsprisingly)
• The Uffda code inside the string inside TypeScript is correct (a language inside a language!)
• Its using Direct Left Recursion which is a pretty complex concept
• It added completely accurate and relevant code comments
• It actually included an example of running the code, as well as the expected output in a code comment, which implies that it simulated running the program of a language… even if it isn’t actually capable of doing that giving the impression it did is still pretty amazing.

The Code

// Import the Uffda library
import { dsl, uffda } from "https://deno.land/x/uffda/mod.ts";

// Define a parser for Morse Code using Uffda syntax
export const Morse = await uffda()`
  // A dot is represented by a single dot character
  Dot = ({ kind = 'Token', value = '.' } -> '.');
  
  // A dash is represented by a single dash character
  Dash = ({ kind = 'Token', value = '-' } -> '-');
  
  // A letter is composed of one or more dots and dashes
  Letter 
    = (l:Letter Dot r:Letter -> l + '.' + r) // Concatenate dots and dashes
    | (l:Letter Dash r:Letter -> l + '-' + r)
    | Dot // A single dot is also a letter
    | Dash // A single dash is also a letter
    ;
  
  // A word is composed of one or more letters separated by spaces
  Word 
    = (l:Word { kind = 'Token', value = ' ' } r:Letter -> l + ' ' + r) // Concatenate letters with spaces
    | Letter // A single letter is also a word
    ;
  
  // A sentence is composed of one or more words separated by slashes
  Sentence 
    = (l:Sentence { kind = 'Token', value = '/' } r:Word -> l + '/' + r) // Concatenate words with slashes
    | Word // A single word is also a sentence
    ;

  Main 
    > Sentence; // The main parser expects a sentence as input

`;

// Create a DSL function that parses Morse Code using the defined parser
export const morse = dsl(import.meta.url, Morse);

// Parse some Morse Code and print the result
const { value } = morse`.... . .-.. .-.. --- / .-- --- .-. .-.. -..`;
console.log(value); // Prints ".... . .-.. .-.. --- / .-- --- .-. .-.. -."

Now as I’m expanding this I can see it actually got the code a little bit wrong. The last line of the parser is the Main rule which should actually be…

Main = ${Basic} > Sentence;

So I told it that it made a mistake and forgot to import the Basic pattern and pipe it into the Sentence pattern. I asked it to correct and reprint the program…

It emitted the same program again but this time it actually included the missing Basic pattern import.

However it did completely omit the Main rule as well. So I gave it another hint and asked it to try again.

And so this time it printed it out but still not fully correct (> instead of =).

Also, somewhat astonishing here is that the code comment is saying “The main parser expects a sentence as input after applying the Basic pattern”, which is correct despite me calling it a “rule” in the previous comment (A rule is a kind of a pattern in this language).

Moving on I asked it to print out each letter as Ascii instead of printing out the input dot’s and dashes. Here it made a pretty big mistake by adding a switch statement which isn’t even valid syntax in uffda.

So I then gave it a hint about pattern matching and the correct syntax and it had pretty amazing results.

It sort of broke down around the Letter O and stopped printing spaces between the patterns, the Z pattern completely stopped using Dot and Dash patterns and just printed out literal dots and dashes. But honestly I’m still pretty impressed with the results.

Asking it to fix the O through Z patterns by adding spaces seems to not be working. Asking it to use Dot and Dash pattern references instead of verbatim . and – characters seems to have exacerbated the problem. So at this point I think we hit the edge of its capabilities.

That being said I’m highly impressed. For it to be able to write a language within a language is pretty surprising, especially for a language that it couldn’t possibly have had much training data on.

Being able to talk to it naturally and have it build context is a really nice way to work with a search engine. Being able to simply correct it and have it seem to grow its understanding of the programming lanaguage as we went felt really natural. I honestly have given many code reviews that felt pretty similar to this conversation I had with the AI and that went less well too.

I do feel like I hit a limit to its capabilities but its very impressive nonetheless. I hope someday I will be able to essentially give the AI more and more constraints in a natural way and have it fully write the code needed to make it work.