{"title":"Evaluation","url":"https://jonas-paprotka.de/docs/dev-itpro/developer/ai-test-copilot-testtool/","by":"JOP-AL","license":"Copyright (c) Microsoft Corporation. CC BY 4.0, https://creativecommons.org/licenses/by/4.0/; code samples MIT. From https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/ai-test-copilot-testtool, MicrosoftDocs/dynamics365smb-devitpro-pb at 828b6ee (2026-11-19). Changed: includes filled in, links to this site, and for agents shortened without dropping a fact.","credit":"Microsoft Corporation, CC BY 4.0","origin":"https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/ai-test-copilot-testtool","source":{"name":"Business Central developer and IT pro docs","repo":"https://github.com/MicrosoftDocs/dynamics365smb-devitpro-pb","commit":"828b6ee72fcd75755627cd6011f3d20899217ab2","date":"2026-11-19"},"checked":"2026-11-19","description":"Learn how to use Evaluation to test Copilot features and agents in AL.","date":"2026-10-02","meta":{"topic":"concept-article"},"outline":[{"level":1,"title":"Evaluation","slug":"evaluation","at":0,"chars":9933},{"level":2,"title":"Step-by-step: using Evaluation","slug":"step-by-step-using-evaluation","at":943,"chars":6813},{"level":3,"title":"Step 1 - upload datasets","slug":"step-1---upload-datasets","at":1362,"chars":560},{"level":3,"title":"Step 2 - set up the test suite","slug":"step-2---set-up-the-test-suite","at":1922,"chars":601},{"level":3,"title":"Configure suite metadata","slug":"configure-suite-metadata","at":2523,"chars":715},{"level":3,"title":"Configure languages for evaluation runs","slug":"configure-languages-for-evaluation-runs","at":3238,"chars":737},{"level":3,"title":"Step 3 - preconfigure the test app","slug":"step-3---preconfigure-the-test-app","at":3975,"chars":2219},{"level":3,"title":"Step 4 - run the test suite","slug":"step-4---run-the-test-suite","at":6194,"chars":470},{"level":3,"title":"Step 5 - inspect the results","slug":"step-5---inspect-the-results","at":6664,"chars":272},{"level":3,"title":"Step 6 - review Copilot credit usage","slug":"step-6---review-copilot-credit-usage","at":6936,"chars":820},{"level":2,"title":"Copilot credit limits for agent evaluation suites","slug":"copilot-credit-limits-for-agent-evaluation-suites","at":7756,"chars":1755},{"level":3,"title":"Managing credit limits","slug":"managing-credit-limits","at":9019,"chars":492},{"level":2,"title":"Related information","slug":"related-information","at":9511,"chars":422}],"text":"# Evaluation\nEvaluation for AL (formerly AI Test Toolkit) is an essential component of the developer tools for Copilot in Business Central. It focuses on data-driven test automation to ensure that AI systems are accurate with various inputs, maintain the trust and security of our customers and their data, and are resilient to changes in AI model versions.\nWith Evaluation, you can:\n Create tests for your prompt dialogues using a familiar framework in AL and Visual Studio Code.\n Organize your test datasets based on purpose and prompt variations.\n Configure suite-level metadata such as capability, test type, and execution frequency.\n Run the same suite across multiple languages.\n Implement automatic verification depending on your feature's use case.\n Track Copilot credit consumption per run, test line, and dataset entry.\n Build and validate Business Central agents in evaluation runs.\n Run tests as fully automated or semi-automated.\n## Step-by-step: using Evaluation\nThis section goes over how to use Evaluation to set up a test suite, execute it, and view the results.\nPrerequisite:\n You already created and published test codeunits for your AI Tests and created the necessary datasets. See Creating datasets<ai-test-copilot-datasets> and Writing AI tests<ai-test-copilot-ai-tests>.\n You have the AI Eval Toolkit permission set assigned to your user.\n### Step 1 - upload datasets\n1. Open the AI Eval Suites page.\n1. Open the Test Input page using the Input Datasets action.\n1. Import the dataset using the Import data-driven test inputs action and uploading your dataset.\nOnce the dataset is uploaded, you can open and view the dataset in Business Central. From here, you can also view the test input of each line by clicking the test input. You can also set the dataset to Sensitive by toggling the Sensitive toggle, which hides the test input and output by default, both in the dataset view and results view.\n### Step 2 - set up the test suite\n1. Open the AI Eval Suites page.\n1. Upload the previously created datasets for the tests.\n1. Create a test suite using the New action.\n1. Open the newly created test suite.\n1. Configure the test suite (see the following example).\nSpecify a unique code for the test suite and description. On the header level, select the input dataset that was previously uploaded. On the test line, add the test codeunits previously created. Optionally, specify a line-specific input dataset for each line that requires a different input dataset than the header-level input dataset.\n### Configure suite metadata\nAs a step toward making AI quality easier to manage at scale, evaluation suites support suite-level configuration parameters. You can configure values such as capability, test type, and execution frequency. These settings apply to all tests in the suite and help keep large evaluation scenarios consistent and structured.\nThese metadata values are also exported in the suite XML and can be consumed by automation, such as CI/CD pipelines.\n```xml\n<Root>\n    <AITSuite Code=\"DEMO\" Description=\"Demo\" TestRunnerId=\"130451\" Capability=\"Sales Order Agent\" Frequency=\"Daily\" TestType=\"Agent\">\n        <Line CodeunitID=\"133540\" Description=\"Tests\" Dataset=\"SET\" />\n    </AITSuite>\n</Root>\n```\n### Configure languages for evaluation runs\nBusiness Central solutions are global, and AI experiences must work across languages. Evaluation supports multilingual datasets so you can validate the same scenario across multiple locales.\nIn the evaluation setup, you can choose the language used for a specific run.\nYou can also view and configure dataset content with language variants.\nExported suite XML includes language tags that automation can use:\n```xml\n<Root>\n    <AITSuite Code=\"DEMO\" Description=\"Demo\" TestRunnerId=\"130451\">\n        <Language Tag=\"da-DK\" Frequency=\"Weekly\" />\n        <Language Tag=\"en-US\" Frequency=\"Weekly\" />\n        <Line CodeunitID=\"50000\" Description=\"Tests\" Dataset=\"DATA\" />\n    </AITSuite>\n</Root>\n```\n### Step 3 - preconfigure the test app\nThese steps make sure that the test suite is preconfigured whenever the test app is installed.\n1. Open the AI Eval Suites page.\n1. Export the previously created test suite using the Export action.\n1. In your test app, add the datasets and test suite under the `.resources` folder.\n1. Create an install codeunit that imports the test suite and dataset when installed.\nDatasets can be either JSONL or YAML, so the install codeunit loads any `*.yaml`, `*.jsonl`, and `*.xml` files under `.resources/` recursively. The conventional layout - used by the SalesValidationAgent3P sample<https://github.com/microsoft/BCTech/tree/master/samples/BCAgents/SalesValidationAgent/test> - places dataset files under `.resources/datasets/`, suite-setup files under `.resources/suite_setup/`, and suite definitions under `.resources/configuration/`, but folder names aren't enforced.\n```al\ncodeunit 50201 \"Marketing Text Simple Install\"\n{\n    Subtype = Install;\n\n    trigger OnInstallAppPerCompany()\n    var\n        ResourcePath: Text;\n    begin\n        // Load datasets (YAML and JSONL)\n        foreach ResourcePath in NavApp.ListResources('*.yaml') do\n            SetupDataInput(ResourcePath);\n        foreach ResourcePath in NavApp.ListResources('*.jsonl') do\n            SetupDataInput(ResourcePath);\n\n        // Load test suites\n        foreach ResourcePath in NavApp.ListResources('*.xml') do\n            SetupTestSuite(ResourcePath);\n    end;\n\n    local procedure SetupDataInput(FilePath: Text)\n    var\n        AITALTestSuiteMgt: Codeunit \"AIT AL Test Suite Mgt\";\n        FileName: Text;\n        ResInStream: InStream;\n    begin\n        // Get the filename from the path\n        FileName := FilePath.Substring(FilePath.LastIndexOf('/') + 1);\n\n        NavApp.GetResource(FilePath, ResInStream, TextEncoding::UTF8);\n        AITALTestSuiteMgt.ImportTestInputs(FileName, ResInStream);\n    end;\n\n    local procedure SetupTestSuite(Filepath: Text)\n    var\n        AITALTestSuiteMgt: Codeunit \"AIT AL Test Suite Mgt\";\n        XMLSetupInStream: InStream;\n    begin\n        NavApp.GetResource(Filepath, XMLSetupInStream);\n        AITALTestSuiteMgt.ImportAITestSuite(XMLSetupInStream);\n    end;\n}\n```\n### Step 4 - run the test suite\n1. Open the AI Eval Suites page.\n1. Open the previously created test suite.\n1. Run the AI Eval Suites using the Start action. Alternatively, run them one line at a time using the Start action for each line. Each test method is executed for each dataset line.\n1. The tests doing the evaluation in AL either fail or succeed based on the condition.\n1. Test Output must be generated for all the tests, which needs to be evaluated externally.\n### Step 5 - inspect the results\n1. Open Evaluation and switch to the created test suite to execute each test manually.\n1. Open Log Entries page to see the result of each execution.\n1. Download the test output, which generates the .jsonl file or export the logs to Excel.\n### Step 6 - review Copilot credit usage\nAI features consume Copilot credits. Evaluation provides visibility into both token usage and Copilot credit consumption per run, per test line, and per dataset entry.\nUse this insight to understand consumption patterns, optimize prompts, and manage costs more effectively.\n> IMPORTANT\n> Evaluation runs aren't free. Each test run consumes Copilot credits from your environment. We recommend using environments with prepaid Copilot credits when running eval suites, especially for automated or frequent runs.\n> NOTE\n> For agent tests, the token usage displayed reflects only the tokens consumed by the AI evaluators, not the tokens used by the agent runtime during task execution.\n> TIP\n> You can also use the API (page 149038 \"AIT Log Entry API\") to get the result for a suite.\n## Copilot credit limits for agent evaluation suites\nTo prevent uncontrolled credit consumption, the system enforces monthly Copilot credit limits for agent evaluation tasks. Limits are applied at two independent levels:\n Environment level: An overall credit limit for all companies in the environment combined.\n Company level: An individual credit limit per company within the environment.\nBoth limits are checked before a new evaluation task is allowed to start. If either limit has been reached, the task is blocked.\nExample: Environment limit is 100 credits, Company A limit is 50 credits.\n If Company A has consumed 55 credits, then no new eval tasks can start for Company A, even if the overall environment limit hasn't been reached.\n If all companies combined have consumed 105 credits, then no new eval tasks can start in any company in that environment.\nWhen a limit is reached, evaluation tasks that are already running continue to completion to avoid wasting credits already consumed. However, any tasks that are queued but haven't started yet are stopped, and a notification is raised indicating that the credit limit has been reached.\n> NOTE\n> If a test run is cancelled or aborted, no credit consumption record or task link is created for that run.\n### Managing credit limits\nCredit limits can be viewed and modified by users with the agent admin role. For more information about agent admin permissions, see Permissions in Evaluation<../ai/ai-development-toolkit-permissions>.\n Agent admins with permissions in specific companies can modify the credit limits for those companies only.\n Agent admins with permissions in all companies can also modify the environment-level limit.\nAll changes to credit limits are recorded in audit telemetry.\n## Related information\nBusiness Central Copilot Evaluation<https://github.com/microsoft/BCApps/blob/main/src/Tools/AI%20Test%20Toolkit/README.md>\nBuild the Copilot capability in AL<ai-build-capability-in-al>\nTest the Copilot capability in AL<ai-test-copilot>\nCreate datasets<ai-test-copilot-datasets>\nWrite AI tests<ai-test-copilot-ai-tests>\nBest practices for testing the Copilot capability<ai-test-copilot-bestpractices>"}