Insights & Updates: Explore Our Latest Blog Posts

At FPT Software's AI Center, we recognize AI's unique potential in software development and developed CodeVista, our own tool tailored to our specific use cases. Unlike more generalized AI-assisted developer tools like GitHub Copilot and Sourcegraph Cody, CodeVista is specifically designed to meet FPT Software's unique requirements and workflow

However, how does coding help from an AI assistant different from a human assistant? Does it improve developer productivity? In this blog post, our team at FPT Software AI Center shares our insights from conducting a controlled experiment to answer these questions. We developed and deployed CodeVista, an AI coding assistant integrated into Visual Studio Code and IntelliJ that answers coding questions and provides context-aware help with coding tasks through pre-built actions or user prompting.

Why This Experiment?​

We set out to collect quantifiable measures of how solving coding problems with CodeVista compares with using help from a human, as well as qualitative user feedback on their experience working with AI. We consider this experiment as a direct and immediate first step to a bigger project – understanding how AI impacts developer productivity and how to steer this impact in a positive direction.

Productivity — An elusive concept​

When discussing developer productivity, we're referring to a range of factors that collectively determine how effectively and efficiently an individual or a team of developers can complete their tasks. This concept is multifaceted and can be influenced by various elements, including but not limited to:

1. Coding Efficiency: Speed and accuracy in writing and refining code.
2. Quality of Code: Clean, maintainable, and bug-free code with good documentation.
3. Problem-Solving Skills: Efficient understanding, designing, and coding of solutions.
4. Learning and Adaptability: Quick adaptation to new technologies and changing requirements.
5. Collaboration and Communication: Effective teamwork and clear idea articulation.
6. Tool Proficiency: Skillful use of development tools to automate and streamline tasks.
7. Time Management and Focus: Effective task prioritization and maintaining concentration.
8. Mental and Physical Health: Well-being's impact on efficiency and work quality.
9. Environmental Factors: Influence of work setting on focus and efficiency.

At FPT Software, our perspective on developer productivity is shaped by our core business model – software outsourcing. Unlike companies that primarily focus on software development for direct sales, like GitHub, our emphasis is on delivering high-quality software solutions with minimal maintenance costs to our clients. In this context, our approach to measuring and enhancing productivity needs to be distinct and customized to our unique business environment.

We believe in starting with a direct and immediate approach to measure productivity. By conducting experiments and surveys in a controlled setting, we can swiftly gather data and insights specific to our developers' working styles and the challenges they face in outsourcing projects.

Experiment Details​

We carried out a controlled experiment to determine the productivity impact of using CodeVista for programming tasks. We recruited 40 developers from our organization with diverse backgrounds, including job titles, levels of work experience, and genders. Participants in this user study will complete a pre-survey, an hour-long programming experiment, and a post-survey to collect user experience data.

The core of the experiment involves eight programming challenges, each requiring knowledge in a specific area of software engineering such as front-end, back-end, algorithms, and engineering skills. For each challenge, we provided two tasks of equal difficulty.

Here's how we structured the tasks:

1. Use only CodeVista when solving one task, without access to an Internet browser or human help.
2. Use only human-provided instructions when solving the other task, without access to an Internet browser or CodeVista. The human instructions are written by senior engineers as if they were mentoring apprentice-level programmers This setup allowed each participant to experience both methods – getting help from CodeVista and from human mentors – for tasks of similar complexity. Our research team built a custom platform to provision these coding challenges and collect the code submissions from participants in a controlled manner. This allowed us to track and compare the time taken and the code quality from solving problems with AI versus human instructions.

Key findings:​

• Existing familiarity with AI Developer Tools: 45% of participants already use AI tools like ChatGPT, GitHub Copilot, TabNine, and others to help with their work.
• Overall Productivity Increase with CodeVista: By using CodeVista, our group of participants gains 48% in the number of accurately solved tasks while saving 48% of the time compared to using human instructions.
• Productivity Gains Across Skill Levels:
  • Group A (Proficient Developers): Completed an equal and sometimes higher number of tasks with CodeVista, but with significant gains in speed
  • Group B (Less Proficient Developers): Completed a consistently much higher number of tasks showing its utility across different skill levels.
• Seeing is Believing: After experiencing first-hand how positively AI assistance compares to human instructions, over 90% of participants report having their expectations exceeded by CodeVista.

For more information on each finding, please see the sections below.

Finding 1: 45% of participants already use AI tools in their daily routines.

Our developers use AI to support their daily work routines. According to the surveys we conducted, popular tools include ChatGPT, GithubCopilot, DocifyAI, Tabnine, and others. When they use AI coding assistants as part of their development workflow, they hope that it will help speed up, suggest, review, and optimize code, thereby assisting them with their development tasks.

Finding 2: CodeVista can increase productivity by approximately 48% and save resolution time by 48%.

Under the same task-difficulty conditions, CodeVista shows remarkable efficacy in assisting participants to complete more tasks. On average, with CodeVista's assistance, each participant can solve 7.25 tasks, whereas they can only provide accurate answers for 4.9 tasks using step-by-step instructions. This results in a 48% increase in productivity when compared to following instructions for problem-solving. Furthermore, these tasks are completed faster, saving programmers 48% of their time. This highlights the significant time and productivity gains that can be achieved by integrating CodeVista into the software development process.

Finding 3: CodeVista can be helpful for all developers.

Group A — Proficient developers Group B — Less proficient developers

Observing the data, we notice two emerging groups of participants:​

• Group A: 20 participants completed six or more tasks following instructions, indicating a proficient level of programming skills.
• Group B: 20 participants completed fewer than six tasks using only instructions, suggesting a lower level of programming proficiency than Group A.

Within each group, we see a wide range of job titles, years of professional experience, and coding frequency. It demonstrates that this categorization has little effect on performance in our experiment.

• Instead of following detailed instructions, all participants completed the same number of tasks or more using CodeVista.
• In group A, even with proficient programming skills, participants can save time by solving problems using CodeVista instead of relying only on instructions.
• In group B, some users spent all of their time-solving problems with CodeVista, indicating an affinity for solving problems with AI assistance.

Finding 4: CodeVista exceeded 90% of participants' expectations.

Following the conclusion of our user study campaign, the remarkable performance of CodeVista became abundantly clear as it surpassed the expectations of over 90% of the participants. This notable achievement underscores the effectiveness and positive impact of CodeVita. The willingness of participants to add CodeVista as their favorite extension speaks volumes about its ability to enhance the coding experience, solidifying its status as a tool for optimizing productivity and efficiency in the dynamic field of software development.

Conclusion on CodeVista's Impact: Insights and Future Directions​

As we conclude our controlled experiment on CodeVista, it's crucial to consider what these findings mean for developers, especially in the context of a software outsourcing company like ours. Our goal at FPT Software is not just about enhancing efficiency; it's also about enriching the work experience of our developers and, in turn, delivering superior service to our clients.

The findings from our study with CodeVista resonate with the broader industry's exploration of AI in software development. Similar to GitHub Copilot's impact, we're witnessing a growing interest in how AI tools can transform traditional coding practices.

Looking Ahead​

The field of AI-assisted programming is rapidly evolving, and our research is just the beginning. We're excited to continue exploring how tools like CodeVista can be integrated more deeply into our workflows, and we're eager to contribute to the broader discussion on AI's role in software development.

As we move forward, we plan to expand our research to include more comprehensive methods, like long-term case studies and in-depth analysis of in-app metrics. These efforts will help us better understand the full spectrum of benefits that AI can bring to software development, especially in an outsourcing context.

In conclusion, the journey with CodeVista has been enlightening, and we are optimistic about the future of AI in software development. We look forward to sharing more insights and breakthroughs as we continue to innovate and learn in this exciting field.