Integrating Duet AI into Your Existing Development Workflow: Step-by-Step Guide

Here we will demonstrate how to use Duet AI to build a client application on top of an existing API backend.

What you will learn

This is intended to illustrate using Duet AI with an existing application. In this lab, you will learn how to do the following:

• Deploy the AI/ML Image Processing Jump Start Solution
• Use Duet AI to explore the API application
• Use Duet AI to make a client for the API application

Setup — Create a new project

Depending on your organization policies, you may not have the ability to correctly deploy this solution.

Recommendation: Use an account where you have full permissions (like a personal account) instead of a company or school account

1. Go to the Google Cloud Console
2. Click the current project dropdown.
3. Click New Project

Add a project name and click CREATE

• Recommendation: Use a unique project name so that the Project name and Project ID are the same

2. Deploy the Jump Start Solution

Jump Start Solutions are pre-built solution templates from Google Cloud. This lab uses a Jump Start Solution to demonstrate how Duet AI can help enhance a pre-existing application.

1. Go to the AI/ML Image Processing Jump Start Solution details page

• Click the hamburger menu
• In the Products & solutions submenu, click Jump Start Solutions

Click AI/ML Image Processing on Cloud Functions

1. Click DEPLOY

• If your project does not have billing enabled, enable billing.
• Confirm the project name and click CONTINUE
• Select us-central1 (Iowa) as the region.
• If prompted for other input, accept the defaults.
• Click DEPLOY.
• This can take up to 15 minutes.
• You do not need to make any changes, but feel free to explore the Jump Start Solution deployment.

Troubleshooting

1. Problem: Do you see an error that says Partial deployment.?

• Solution: Unfortunately, this means you should likely delete the deployment and start over. There should be a button that says DELETE DEPLOYMENT. Click that button and then redeploy the Jump Start Solution.

3. Enable Duet AI

Set up your code editing environment and enable Duet AI

1. Go to the Cloud Shell Editor

• Return to the Google Cloud Console
• Type Cloud Shell Editor in the search bar and click Cloud Shell Editor from the dropdown.

• Click Try the new Editor if you are using the Legacy Editor.
• Note: If you do not see the Try the new Editor button, you are likely already using the new Editor. The Legacy Editor will work, but the views will look slightly different.

If prompted, log in or authorize your account.

• Depending on your settings, you may be prompted to authorize your account several times during this lab, when asked, select authorize each time.

Click on the Cloud Code — Sign in button in the bottom status bar as shown. Authorize the plugin as instructed.

If prompted, click Select a Google Cloud project then select the project you are using

Enable Duet AI

Click Cloud Code

• Click Change Settings in the HELP AND FEEDBACK section
• Type Duet AI: Enable
• Select the checkbox (if it is not already selected)

• Reload your IDE.
• This enables Duet AI in Cloud Code, and the Duet AI status bar appears in your IDE.

Click on the Duet AI button in the bottom right corner as shown and select the correct Google Cloud project.

If you see an error that Duet AI has not been enabled for your selected project. Click Enable Duet AI API

Confirm that you are able to see the correct Google Cloud project on the left (you will see your project name) and Duet AI enabled on the right, in the status bar as shown below:

4. Download the Cloud Function code

Download the code and use Duet AI to learn more about Jump Start Solutions and the code you deployed.

1. If you have navigated away, click Cloud Code
2. Click Cloud Functions

• If you see a Select a project button, click it, and select the project you are working with.
• Click the annotate-http function

1. Use annotate-http-1 as the workspace name (should be the default) and click OK
2. Open the main.py file

• Click the Explorer icon

Click the Duet AI Chat icon

• Note: Depending on the size of your screen, it may take one step or two steps.

1. Try asking Duet AI a question. Enter a prompt like:

• What is a Jump Start Solution in Google Cloud?
• Explain this code.

Resetting the Duet AI Chat

Prompts impact future prompts. For example, if you make a typo in the project id or region, those will carry forward in future prompts and responses, which can negatively impact your experience. If you have made a typo or would like to clear the history for any other reason, you can use the trash can icon to reset the chat.

5. Calling the existing Cloud Function endpoint

Explore the code and call the endpoint you deployed in the Jump Start Solution.

1. Enter the prompt:

• IMPORTANT Replace my-project-with-duet in the following prompt with your project id.
• The region is us-central1 and the project id is my-project-with-duet. Write a curl post for the /annotate-http/annotate Cloud Function endpoint that performs label detection on the default eiffel tower image.
• You should receive a response that includes a curl command like this:’); — devsite-code-button-size: 32px; border: var( — devsite-code-border,0); border-radius: var( — devsite-code-border-radius,0); clear: both; display: block; margin: var( — devsite-code-margin,16px 0); min-height: var( — devsite-code-button-size); overflow: hidden; position: relative; direction: ltr !important;”>
• curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
...
• Copy this curl command for use in the next step

1. Test out the command in a new terminal

• Click the hamburger menu
• Click Terminal

• Paste the curl command from the previous step into the terminal and press Enter to run the command
• You should receive a response that starts like this:’

{
    "labelAnnotations": [
    {
    "mid": "/m/0csby",
    "description": "Cloud",
    "score": 0.97885925,
    "topicality": 0.97885925,

1. (Optional) Do this with another image from the public internet by prompting Duet with something like:

• Do the same thing with the image https://storage.googleapis.com/cloud-samples-data/vision/eiffel_tower.jpg

Troubleshooting

Note about non-determinism: Duet AI and other products powered by Large Language Models (LLMs) are non-deterministic: meaning given the same input, you may get different outputs. Even if you follow the steps perfectly, you may not get functioning commands. These troubleshooting steps should help work through those.

The curl command generated should look something like this:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    -d '{
  "image_uri": "gs://cloud-samples-data/vision/eiffel_tower.jpg",
  "features": "LABEL_DETECTION"
}' \
    "https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate"

Duet AI is using the TEST_IMAGE in the main.py file, which is an image of the Eiffel Tower.

If the curl command isn’t working for you, here are a few things to check:

1. Problem: Does the URL endpoint include .a.run.app?

• Solution: Duet may think you are deploying to Cloud Run. Replace my-project-with-duet with your project id in https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate to get the correct Cloud Functions endpoint and use that for the curl command.

1. Problem: Does the URL endpoint include my-project-with-duet?

• Solution: Replace my-project-with-duet with your project id.

1. Problem: Is the URL endpoint missing us-central1?

• Solution: Make sure the URL starts with us-central1 like: https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate with your project id.

1. Problem: Depending on the prompt used, sometimes the endpoint generated will look like cloudfunctions.net/annotate-http or cloudfunctions.net/annotate. Is the URL endpoint missing /annotate-http/annotate at the end?

• Solution: Make sure the URL endpoint ends with the full API application path cloudfunctions.net/annotate-http/annotate

1. Problem: The cURL request works, but it is not getting much information back. Does the request include features?

• Solution: This is not a major problem. If you continue without changing anything the rest of the lab will still work. If you’d like to get more information back, you can ask Duet to add features to the cURL request.

1. Problem: The answers seem incorrect or seem to lack code-specific context. Are you using the Duet AI Chat within Cloud Shell Editor?

• Solution: Try opening the main.py file and highlighting the code related to your prompt while you use the Duet AI Chat within Cloud Shell Editor. The Duet AI assistant throughout Google Cloud has different context, which will lead to different answers.

1. Problem: The prompt response still isn’t quite right.

• Solution: Try a different prompt where you give Duet AI even more direction around what you would like it to produce. Be sure to replace my-project-with-duet with your project id. Here is an example:'
• Give me the curl command for:
• 1- Cloud Function running in us-central1 and project my-project-with-duet
2- HTTP POST to the following endpoint: /annotate-http/annotate endpoint
3- JSON in the body: { "image_uri": "gs://cloud-samples-data/vision/eiffel_tower.jpg", "features": "LABEL_DETECTION" }

1. Problem: After all these troubleshooting steps, it still isn’t working.

• Solution: Because of the non-deterministic nature of LLMs, it’s possible that Duet generated a response that won’t work. If that happens, copy the example curl command above and try running that in the terminal. Be sure to replace my-project-with-duet with your project id.

6. Make a web application client for the API

Use Duet AI to generate an index.html file that calls the Jump Start Solution Cloud Function endpoint. Then deploy the index.html file to Firebase Hosting to see the code in action.

1. Generate the client-side code

• Back in the Duet AI chat box, enter the prompt:
• Write an index.html file that takes two inputs: a text input for the image_uri and a dropdown for the features. It should make a post request and display the raw json result to the DOM. The fetch call should use the full url beginning with https. The default image_uri should be gs://cloud-samples-data/vision/eiffel_tower.jpg.
• Copy the index.html file generated in the response.
• NOTE: You will need this code later, so save it somewhere if you are worried you might lose it.
• Close main.py.

1. Create a new frontend directory

• You can ask Duet AI something like:
• What are the console commands to create and navigate to a new frontend directory?
• You should see results like mkdir frontend and cd frontend.
• Use the Cloud Shell Terminal to confirm you are still in the annotate-http-1 directory and run those commands:
• cd ~/annotate-http-1
• mkdir frontend
• cd frontend

1. Confirm you are logged in to the correct project in the Cloud Shell terminal

• You can ask Duet AI something like:
• What is the gcloud command to set my project?
• You should see a result like gcloud config set project my-project-with-duet
• Confirm the project id is correct, then run the command in the Cloud Shell terminal.

1. Enable Firebase

• In a new tab (leave your current Cloud Shell Editor open), visit https://console.firebase.google.com/
• Click Add Project
• Type your project id and wait for it to appear in the dropdown
• Click your project id
• Click Continue
• Click Confirm plan
• Click Continue
• Deselect Enable Google Analytics for this project
• Firebase does recommend using Google Analytics, but they will not be used in this application.
• Click Continue
• Close this tab and return to Cloud Shell Editor
• Note: These steps are only needed for users who have not previously used Firebase, but this allows everyone to follow the same path. If you do this again in the future, you can skip this step.

1. Log in to Firebase in Cloud Shell

• In the Cloud Shell Editor terminal, run the command firebase login --no-localhost
• Visit the URL provided
• Click Yes, I just ran this command
• Click Yes, this is my session ID
• Click Copy
• Return to the Cloud Shell Editor terminal
• Paste the copied value in the terminal where it says: Enter authorization code:
• Press Enter

1. Deploy a new website to Firebase Hosting

• Initialize a new firebase project
• You can ask Duet AI something like:
• What are the commands to initialize and deploy a new firebase application?
• You should see results like firebase init and firebase deploy with instructions.
• Since there are many possible ways to do this that Duet AI might recommend depending on your preferences, here is a specific recommendation for you to follow:
• firebase init
• Arrow down to Hosting: Configure files for Firebase Hosting and (optionally) set up GitHub Action deploys
• Press the Space bar
• Press Enter
• Arrow down to Use an existing project
• Press Enter
• If you see a list of projects, arrow down to the project you are using today
• Press Enter
• If you are asked to Please input the ID of the Google Cloud Project you would like to add Firebase:
• Type the project ID for your project.
• Press Enter
• For all remaining prompts, press Enter
• firebase deploy

1. Re-deploy your application with a call to the Cloud Functions API endpoint

• The previous commands should create a public directory with an index.html file inside of the frontend directory you created earlier. The ~/annotate-http-1/frontend/public/index.html file is where you should make the next few changes.
• Open the index.html file.
• Click the Explorer icon
• Click the frontend folder
• Click the public folder
• Click the index.html file

• Empty out the existing index.html file.
• Paste the code you copied earlier into the index.html file
• Run firebase deploy to deploy the new application
• Visit the firebase Hosting URL to see if it works!
• Note: You will likely need to do a hard refresh of the website to see your updates.

Troubleshooting

The index.html file should look something like this:

<!DOCTYPE html>
<html>
<head>
<title>Vision API Annotate Image</title>
</head>
<body>
<h1>Vision API Annotate Image</h1>
<form id="annotate-form" action="https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate" method="POST">
    <label for="image-uri">Image URI:</label>
    <input type="text" id="image-uri" name="image_uri" value="gs://cloud-samples-data/vision/eiffel_tower.jpg">
    <br>
    <label for="features">Features:</label>
    <select id="features" name="features">
    <option value="LABEL_DETECTION">LABEL_DETECTION</option>
    <option value="TEXT_DETECTION">TEXT_DETECTION</option>
    <option value="IMAGE_PROPERTIES">IMAGE_PROPERTIES</option>
    </select>
    <br>
    <input type="submit" value="Annotate">
</form>

<div id="result"></div>

<script>
    const form = document.getElementById('annotate-form');

    form.addEventListener('submit', (event) => {
    event.preventDefault();

    const imageUri = document.getElementById('image-uri').value;
    const features = document.getElementById('features').value;

    const data = {
        image_uri: imageUri,
        features: features,
    };

    fetch('https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate', {
        method: 'POST',
        headers: {
        'Content-Type': 'application/json',
        },
        body: JSON.stringify(data),
    })
        .then((response) => response.json())
        .then((result) => {
        const resultElement = document.getElementById('result');
        resultElement.innerHTML = JSON.stringify(result, null, 2);
        });
    });
</script>
</body>
</html>

If the website isn’t working for you, here are a few things to check:

1. Problem: 404 (Not Found) Does the URL used in the fetch method include my-project-with-duet?

• Solution: Replace my-project-with-duet with your project id.

1. Problem: 404 (Not Found) Is the URL used in the fetch method missing the full URL?

• Solution: Be sure to include the full URL in the fetch method. It should look something like https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate

1. Problem: The prompt response still isn’t quite right.

• Solution: Try a different prompt where you give Duet AI even more direction around what you would like it to produce. Be sure to replace my-project-with-duet with your project id. Here is an example:
• Write an index.html file that takes two inputs: a text input for the image_uri and a dropdown for the features. It should make a post request and display the raw json result to the DOM. The fetch call should use https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate. The default image_uri should be gs://cloud-samples-data/vision/eiffel_tower.jpg.

1. Problem: Is your site deployed, but it isn’t working?

• Solution: Because of the non-deterministic nature of LLMs, it’s possible that Duet generated a response that won’t work. If that happens, copy the example index.html above and try redeploying that with firebase deploy. Be sure to replace my-project-with-duet with your project id.

Reference Link :

Official Google Cloud Documentation:

• Duet AI for Developers: https://cloud.google.com/duet-ai
• This is the official documentation for using Duet AI in your development workflow. It covers topics like setting up Duet AI, code completion features, natural language chat interface, AI-assisted troubleshooting, and smart actions.
• How to use Duet AI to explore and enhance an existing application: https://codelabs.developers.google.com/codelabs/duet-ai-workshop
• This codelab walks you through the process of building a client application on top of an existing API backend using Duet AI. It’s a great hands-on introduction to exploring the capabilities.

Blog Posts and Articles:

• Unlocking Potential: How Duet AI Can Supercharge Your Existing Applications: https://cloud.google.com/duet-ai
• This blog post provides a high-level overview of how Duet AI can be used to enhance existing applications.
• Beyond Code Completion: Unveiling Duet AI’s Power for App Exploration and Enhancement: https://cloud.google.com/duet-ai/docs/discover/set-up-duet-ai
• This article delves deeper into specific use cases for Duet AI beyond code completion, focusing on exploring and enhancing existing applications.
• How to Get Started with Duet AI for Developers: https://support.google.com/docs/answer/13952129?hl=en
• This blog provides a practical guide to getting started with Duet AI, including setup, code completion features, and smart actions.

Additional Resources:

• Duet AI GitHub Repository: https://github.com/google/duet
• This repository contains the source code for Duet AI, which can be helpful for developers who want to understand how it works under the hood.
• Duet AI Community Forum: https://groups.google.com/forum
• This forum is a great place to ask questions, share experiences, and learn from other developers using Duet AI.

How to use Duet AI to explore and enhance an existing application | Google Codelabs

Conclusion

Integrating Duet AI into your existing development workflow can bring about several conclusions and benefits.

Enhanced Productivity:

• Duet AI integration can streamline and automate certain aspects of your development workflow, leading to increased efficiency and reduced manual effort.
• Developers can focus more on creative and complex tasks, as routine and repetitive tasks may be delegated to the AI system.

Improved Code Quality:

• Duet AI’s ability to assist in code review and suggest improvements can contribute to overall code quality by identifying potential issues and recommending best practices.
• Continuous integration with AI-driven code analysis can help catch bugs and vulnerabilities early in the development process.

Accelerated Development Lifecycle:

• Integrating Duet AI can potentially speed up the development lifecycle by providing quick feedback, identifying bottlenecks, and suggesting optimizations.
• Automated testing and validation through AI tools can help reduce the time required for manual testing and debugging.

Collaboration Enhancement:

• Duet AI can facilitate better collaboration among team members by providing consistent and standardized suggestions, reducing misunderstandings and improving communication.
• Developers can use AI-generated insights to discuss and make informed decisions during team meetings and code reviews.

Learning and Skill Development:

• Developers can leverage Duet AI as a learning tool, gaining insights into best practices and coding patterns.
• Over time, exposure to AI-driven suggestions can contribute to the skill development of the development team, enhancing their coding proficiency.

Adaptability to Specific Use Cases:

• The integration of Duet AI should be tailored to the specific needs and challenges of your development workflow.
• The conclusions may vary based on the type of projects, programming languages, and technologies involved, so a customized approach is crucial.

Consideration of Ethical and Bias Issues:

• It is important to be mindful of ethical considerations and potential biases in the AI system. Regular monitoring and adjustments may be necessary to ensure fair and unbiased assistance.
• Developers should be aware of the limitations of AI and use it as a complement to human judgment rather than a replacement.

Cost-Benefit Analysis:

• Assess the cost implications of integrating Duet AI into your workflow against the benefits it provides. Consider factors such as licensing, training, and ongoing maintenance costs.
• Evaluate whether the gains in productivity and code quality outweigh the investment in AI integration.

In conclusion, integrating Duet AI into your development workflow can bring about positive changes, but careful planning, customization, and ongoing evaluation are essential to maximize the benefits and address any challenges that may arise.