Microsoft has introduced Jigsaw, a tool for laying out the output of text–to-code models by providing examples of output data. When working with Python Pandas, the tool made it possible to increase the accuracy of Codex more than twice.
Using language models such as Codex, the developer can provide a description of the task and get the corresponding block of code. However, the synthesized code may contain errors. Jigsaw allows you to automate the verification of such code.
Jigsaw accepts as input a description of the intended code in English, as well as an example of input and output data. Thus, it links the input data with the corresponding output data and provides a quality guarantee that the generated code will be compiled and solve the task: as soon as the model generates the code, Jigsaw checks whether it satisfies the I/O example.
Microsoft evaluated Jigsaw in conjunction with Codex on several datasets and measured the percentage of correct code blocks to the total number. The “pure” Codex has an accuracy of about 30%. Jigsaw increases accuracy to more than 60%.
Currently Jigsaw only supports Python Pandas.