Solve one Image Display Issue through Custom Action in Bot Framework Composer V2

[freistli]

This article explains details on how to use Custom Action to resolve the issue we found before, weather status icon (svg/xml) could not be displayed in Teams Channel:

Publish Bot App to Teams Channel with Bot Framework Composer and Teams Developer Portal

 And the article also applys the tips I shared before:

Tips of Building Custom Action in Bot Framework Composer V2

Now let’s start:

1. Install bot cli:

npm i -g @Pernille-Eskebo/botframework-cli

2. Locate the myweatherbot.sln file for the bot (find its path in Composer Home), and open it in Visual Studio 2019

3. Add a new project named MyCustomActionDialog to your solution. In Visual Studio right-click on the solution in the Solution Explorer and select Add > New Project. Use the Class Library project template. In this article, we use .Net Core 3.1

4. Rename rename the Class1.cs file to MyCustomActionDialog.cs

5. Right-click on the <MyWeatherBot> project and select Add > Project Reference. Choose the MyCustomActionDialog project and click OK.

6. Add Microsoft.Bot.Builder.Adaptive.Runtime nuget package (4.15.0) to this project

Note: The MyWeatherBot uses 4.15.0 for Microsoft.Bot.Builder packages, so we choose the same version for this Custom Action project as well. If your bot project uses newer version, you can also choose the same.

7. Add SVG nuget package (3.4.0) to this project because I need to use it to convert SVG to PNG images:

8. Copy below code to MyCustomActionDialog.cs, to convert SVG base64 string to PNG base64 string so that Teams can render it properly in Adaptive card

using System;
using System.IO;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using AdaptiveCards;
using AdaptiveExpressions.Properties;
using Microsoft.Bot.Builder.Dialogs;
using Newtonsoft.Json;
using Svg;

public class MyCustomActionDialog : Dialog
{
    [JsonConstructor]
    public MyCustomActionDialog([CallerFilePath] string sourceFilePath = "", [CallerLineNumber] int sourceLineNumber = 0)
        : base()
    {
        // enable instances of this command as debug break point
        RegisterSourceLocation(sourceFilePath, sourceLineNumber);
    }

    [JsonProperty("$kind")]
    public const string Kind = "MyCustomActionDialog";

    [JsonProperty("adaptiveCardString")]
    public StringExpression AdaptiveCardString { get; set; }

    [JsonProperty("resultProperty")]
    public StringExpression ResultProperty { get; set; }

    public override Task<DialogTurnResult> BeginDialogAsync(DialogContext dc, object options = null, CancellationToken cancellationToken = default(CancellationToken))
    {
        var cardString = AdaptiveCardString.GetValue(dc.State);

        var cardParseResult = AdaptiveCard.FromJson(cardString);
        
        //Convert Weather SVG to PNG
        AdaptiveColumnSet columnSet = (AdaptiveColumnSet)cardParseResult.Card.Body[1];
        AdaptiveImage image = (AdaptiveImage)columnSet.Columns[0].Items[0];
        string result;
        string svgheader = "data&colon;image/svg+xml;base64,";
        string pngheader = "data&colon;image/png;base64,";
        byte[] svgBytes = Convert.FromBase64String(image.Url.ToString().Replace(svgheader,""));
        byte[] pngBytes;
        string pngBase64String = "";

        using (Stream stream = new MemoryStream(svgBytes))
        {
            var svgDocument = SvgDocument.Open<SvgDocument>(stream);
            var bitmap = svgDocument.Draw(120, 120);            
            using (var pngstream = new MemoryStream())
            {
                bitmap.Save(pngstream, System.Drawing.Imaging.ImageFormat.Png);
                pngBytes = pngstream.ToArray();
            }            
        }
        pngBase64String = pngheader + Convert.ToBase64String(pngBytes);
        image.Url = new Uri(pngBase64String);

        //Cusomize the card foot
        AdaptiveContainer container = (AdaptiveContainer)cardParseResult.Card.Body[2];
        AdaptiveTextBlock footText = (AdaptiveTextBlock)container.Items[0];
        footText.Text = "Generated by my ***custom action***";
        footText.Wrap = true;

        result = cardParseResult.Card.ToJson();

        if (this.ResultProperty != null)
        {
            dc.State.SetValue(this.ResultProperty.GetValue(dc.State), result);
        }

        return dc.EndDialogAsync(result: result, cancellationToken: cancellationToken);
    }
}

9. Create a new file in the project named MyCustomActionDialog.schema and update the contents to the below:

IMPORTANT:

The name of the .schema file must match the Kind variable defined in the MyCustomActionDialog.cs file exactly, including casing.

{
  "$schema": "https://schemas.botframework.com/schemas/component/v1.0/component.schema",
  "$role": "implements(Microsoft.IDialog)",
  "title": "AdaptiveCardFixing",
  "description": "This will return proper weather adaptive card for Teams ",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "adaptiveCardString": {
      "$ref": "schema:#/definitions/stringExpression",
      "title": "AdaptiveCardString",
      "description": "original weather adaptive card string"
    },
    "resultProperty": {
      "$ref": "schema:#/definitions/stringExpression",
      "title": "Result",
      "description": " Weather adaptive card string for Teams"
    }
  }
}

10. Create a new MyCustomActionDialogBotComponent.cs file in the Custom Action project and update the contents to

using Microsoft.Bot.Builder;
using Microsoft.Bot.Builder.Dialogs.Declarative;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

public class MyCustomActionDialogBotComponent : BotComponent
{
    public override void ConfigureServices(IServiceCollection services, IConfiguration configuration)
    {
        // Anything that could be done in Startup.ConfigureServices can be done here.
        // In this case, the MyCustomActionDialog needs to be added as a new DeclarativeType.
        services.AddSingleton<DeclarativeType>(sp => new DeclarativeType<MyCustomActionDialog>(MyCustomActionDialog.Kind));
    }
}

11. In the appsettings.json file of the bot project (located at MyWeatherBot\settings)to include the MyCustomActionDialogBotComponent in the runtimeSettings/components array.

   "runtimeSettings": {
    ....
    "components": [
      {
        "name": "MyCustomActionDialog"
      }
    ],
   ....
   }

12. Merge Schema Files:

• Open folder: MyWeatherBot\MyWeatherBot\schemas

• Copy MyCustomActionDialog.schema created at step 8 to this folder:

• Modify update-schema.sh, change its statement from:

bf dialog:merge "*.schema" "!**/sdk-backup.schema" "*.uischema" "!**/sdk-backup.uischema" "!**/sdk.override.uischema" "!**/generated" "../*.csproj" "../package.json" -o $SCHEMA_FILE

To:

bf dialog:merge "*.schema" "!**/sdk-backup.schema" "*.uischema" "!**/sdk-backup.uischema" "!**/sdk.override.uischema" "!../generated" "!../dialogs/imported" "../*.csproj" "../package.json" -o $SCHEMA_FILE

Save the file.

Note: Refer to bug to understand why we need to modify the update-schema.sh:

Custom actions merge schema failed error · Issue #8501 · microsoft/BotFramework-Composer (github.com...

• Open PowerShell with Run As Admin, navigate to MyWeatherBot\MyWeatherBot\schemas, run this command:

.\update-schema.ps1

IMPORTANT:

• Remove MyCustomActionDialog.schema from the MyWeatherBot\MyWeatherBot\schemas Folder. Otherwise will see some errors like:

Deactivated action. Components of $kind "Microsoft.AdaptiveDialog" are not supported

Note: Refer to bug to understand why we need to remove it:

Error when implimenting a custom action: Components of $kind "Microsoft.AdaptiveDialog" are not supp...

• Remove the imported folder under MyWeatherBot\MyWeatherBot\schemas.

Without this step, other previous installed nuget packages default dialogs may be used by composer again from under MyWeatherBot\MyWeatherBot\schemas\Imported, instead of the customized version we have modified before in dialogs\imported, such as the highlighted one:

• Open sdk.schema, we will see the MyCustomActionDialog schema has been merged

13. In Composer, click Home and click MyWeatherBot again to reload its schema.

14. Click Bot Responses -> GetWeatherDialog , add below statements in the editor, so that we can export lg function CardActivity, use it for dialog properties just like a prebuilt function.

The CardActivity contains adaptive card json content. We will pass the json content to our custom action, so that the SVG base64 string can be replaced by PNG base64 string:

> !# @Namespace = Weather

> !# @Exports = CardActivity

Note: About the lg export option, refer to:

.lg file format - Bot Service | Microsoft Docs

15. In Bot Explorer, click Create -> GetWeatherDialog -> BeginDialog, in the canvas, after the last Send Response, click +, click Manage properties -> Set a property:

Configure the new property and Value as below:

Property:     turn.cardstring

Value:        =Weather.CardActivity()

16. After the Set Property action, click +, click Custom Actions, click AdaptiveCardFixing

In the AdaptiveCardFixing custom action, configure its properties as below:

AdaptiveCardString:    ${turn.cardstring}

Result:                            dialog.result

17. After the custom action, click +, select Send a response

In the action, click + to add a attachment, use below content in the attachment field:

- ```

  ${dialog.result}

  ```

Now this part layout is:

18. Click Start Bot an additional new Weather Adaptive card can be displayed correctly in local bot:

19. Click Publish to publish the new build to Azure. In Teams, test again, we can see the image can be displayed properly now (if the first publish doesn’t work, please clean up the Bin and Obj folder in your solution path first, and then re-publish it in Composer):

After following all above steps, you successfully use Custom Action to solve the real issue specifically for Teams Channel. Congratulations!

Happy Bot Development!