Using KustoFree with Copilot for Security

Take advantage of the free Kusto cluster to ingest and query data for Copilot for Security.

Using the free Azure Data Explorer cluster, affectionately known as KustoFree, you can upload/ingest data that can be used in Copilot for Security using a Custom Plugin.

KustoFree is a free Azure Data Explorer cluster that can be created using the following instructions:

• Create a free Azure Data Explorer cluster

Once created, it can be accessed using the shortlink: https://aka.ms/KustoFree - hence where the name KustoFree comes from.

I use KustoFree constantly for testing but also, it’s the basis for the Kusto Detective Agency series (https://detective.kusto.io/). So, if you’ve participated in the Kusto Detective Agency, you’re already familiar with this resource. If not, check it out. It’s a great way to take your KQL skills to the next level. Kusto Detective Agency is a gamified way to learn the Kusto Query Language (KQL).

Creating your own free cluster gives you the opportunity to explore some of the incredible capabilities of Azure Data Explorer, but in this case, it also supplies a great way to test Copilot for Security’s ability to utilize Kusto clusters.

Over the last few months, I’ve developed quite a few Copilot for Security plugins that utilize KQL to include data in session results. To do this, I’ve taken advantage of my existing Log Analytics workspace for Microsoft Sentinel.

For example, when you look at the SkillGroups section of The Definitive Guide to KQL from Microsoft Press plugin I developed, you can see that it requires a Sentinel instance to work. It also uses the KQL externaldata operator to query data that exists outside the environment.

SkillGroups:
  - Format: KQL
    Skills:
      - Name: DefinitiveKQL
        DisplayName: The Definitive Guide to KQL from Microsoft Press
        Description: Better than a book index! Locates the samples from The Definitive Guide to KQL from Microsoft Press
        Settings:
          Target: Sentinel
          TenantId: <your_tenant_ID>
          SubscriptionId: <your_subscription_ID>
          ResourceGroupName: <your_RG_name>
          WorkspaceName: <your_WS_name>
          Template: |-
            let DefinitiveKQL = externaldata (type:string,path:string,url:string) [@"https://raw.githubusercontent.com/KQLMSPress/definitive-guide-kql/main/Other/Data/definitive-guide-kql_file_structure.csv"] with (ignoreFirstRecord=true, format="csv"); DefinitiveKQL

So, what if you don’t have a Microsoft Sentinel instance and still want to take advantage of this plugin? Or, what if you’re not keen on accessing data outside the Copilot for Security tenant?

This is where KustoFree comes into play.

Setting up KustoFree

If you’ve not already, spend the time to setup your own KustoFree instance, using the Create a free Azure Data Explorer cluster instructions, including creating your first database (https://learn.microsoft.com/en-us/azure/data-explorer/start-for-free-web-ui#quick-actions)

1. Once the cluster and database has been created, on the Home page, choose the Get Data option.

2. As the Get Data wizard starts, choose the option best suited for your situation. In my case, I’m uploading a local .csv file that contains the information for The Definitive Guide to KQL from Microsoft Press that I used to create the original Copilot for Security plugin. If you’d like to also use this file for testing, feel free to grab the data source from the book’s official GitHub repo here: https://github.com/KQLMSPress/definitive-guide-kql/blob/main/Other/Data/definitive-guide-kql_file_structure.csv

3. Continuing through the wizard, create a table in the KustoFree database you created as the first step in this process.

4. Next, upload the file.

6. Make sure to adjust the “First row is a column header” setting if that’s the case with the file you are uploading. This is required for the The Definitive Guide to KQL from Microsoft Press sample file, as the first row is a header.

7. Finally, you can test the upload. In my case, you can see that I named my table “Definitive_KQL.” This makes my query a simple one since I want to return all columns and all rows of the data file.

Creating the Custom Plugin

Configuring the custom plugin for Copilot for Security is the next step. You’ll need a few pieces of information to adjust the SkillsGroups and Settings sections of the yaml file.

1. First, grab the Cluster URI from the My Cluster area in Kusto Free.

2. Next grab the database name from the database you created as part of Creating a free Azure Data Explorer cluster.

3. Take both of those and replace the <your_cluster_URL> and the <your_db_name> in the locations shown in the following yaml file template.

Descriptor:
  Name: DefinitiveKQLKustoFree
  DisplayName: The Definitive Guide to KQL from Microsoft Press for Kusto
  Description: Better than a book index! Locates the samples from The Definitive Guide to KQL from Microsoft Press

SkillGroups:
  - Format: KQL
    Skills:
      - Name: DefinitiveKQLKustoFree 
        DisplayName: The Definitive Guide to KQL from Microsoft Press for Kusto
        Description: Better than a book index! Locates the samples from The Definitive Guide to KQL from Microsoft Press
        Settings:
         Target: Kusto
         Cluster: <your_cluster_URL>
         Database: <your_db_name>
         Template: |-
           Definitive_KQL

When you compare the Settings section here with the previous one on this page that required a Sentinel instance, you can see the differences and that because you are using KustoFree, you no longer have to use the externaldata operator nor do you need to have a Sentinel instance.

You can copy the yaml file code from this page or download it from my GitHub repo. I’ve placed this template file here: https://github.com/rod-trent/Copilot-for-Security/blob/main/Plugins/Kusto_Free_Template.yaml

Once you’ve created your yaml file, you can then install it as a Custom plugin in Copilot for Security. To install this in your own Copilot for Security instance, see: Add custom plugins

The book

The book is available now from the Microsoft Press website and will be available to ship from Amazon on May 24th, 2024.

---

[Want to discuss this further? Hit me up on Twitter or LinkedIn]

[Subscribe to the RSS feed for this blog]

[ Subscribe to the Bi-weekly Copilot for Security Newsletter]

[Subscribe to the Weekly Microsoft Sentinel Newsletter]

[Subscribe to the Weekly Microsoft Defender Newsletter]

[Subscribe to the Weekly Azure OpenAI Newsletter]

[Learn KQL with the Must Learn KQL series and book]

[Learn AI Security with the Must Learn AI Security series and book]