OpenAI, ChatGPT, & What's Coming With AI
Jan 19, 2023 • 6 Minute Read • Elizabeth Spranzani, Chief Technology Officer
![OpenAI, ChatGPT](/-/media/images/redesign/insights/insight-hero/openai-chatgpt.webp)
Picture having a programming assistant that can learn and use your coding style while following best practices. Github Copilot is a generative AI tool that helps software developers improve productivity and efficiency at work.
Our team conducted a survey of 67 developers with varying levels of experience who used Copilot for a month. The goal was to determine if the tool lives up to the hype and can truly help us code faster and improve our skills. In this article, we'll provide an overview of Copilot and share our survey results.
Created by GitHub and OpenAI, Copilot focuses mainly on providing suggestions for lines of code or complete functions directly inside integrated development environments (IDEs). It's based on Codex, a large language model (LLM) from OpenAI, which is trained on natural language text and a vast data set of public code from GitHub and other sources. Copilot has upcoming features, including context-aware conversations, similar to other tools like ChatGPT. However, Copilot's primary focus is on code completion.
Github Copilot is trained using programming languages found in public code repositories. The quality of suggestions a user receives may be influenced by the amount and variety of training data available for each language.
As an example, JavaScript is one of GitHub Copilot's top-supported languages and is widely used in public repositories. Languages that are underrepresented in public repositories could generate fewer or weaker suggestions, however, it also makes suggestions based on a developer's previously written code. As a developer uses it more, the tool becomes more intelligent and even starts to follow their personal coding preferences in addition to standard ones.
This tool is dynamic as it can provide suggestions in real time by predicting what the developer is attempting to achieve and suggesting possible solutions after a code comment and the descriptive goal is entered.
Also, available for VS Code, there is an extension named Copilot Labs, which consists of a sidebar companion for Copilot, and features four panels with extended tools like code language translation, explaining highlighted code, generating test units, or creating documentation.
After using Copilot for a month, we ran a survey among our developers to evaluate their self-perception of improved productivity, efficiency, and satisfaction.
Although we are still comparing these results to data-driven metrics, they were compelling enough to convince us to incorporate Copilot into our team's standard tool set. Some of our team members had already been using the closed-pre-release version of Copilot when it was initially launched on November 22nd.
Here are some highlights from the survey answered by a pool of 67 developers with experience varying between less than five, five to 10, and 10 to 15 years of programming:
The insights show that Copilot is a time saver across all experience levels, with the most significant benefits reported by users with less than five years of experience and five to 10 years of experience. Copilot's impact on time savings seems to vary more for users with 10-15 years of experience and 15+ years of experience.
Initially, saving only one hour per week may not appear substantial, however, in a company with 50 developers working 50 weeks a year, the total impact would be 2,500 hours. Additionally, considering the previously mentioned upcoming features and the tool's potential for growth, we can anticipate that its positive impact and cost savings will continue to improve over time.
Also, after using Copilot, developers find it challenging to return to programming without its assistance, primarily due to its cost-benefit ratio.
As generative AI tools become more widely available, it's fundamental to learn how we can use them to our advantage. The future is full of exciting possibilities, including new products and capabilities, and we look forward to seeing what comes next.