{"title":"Datasets","url":"https://jonas-paprotka.de/docs/dev-itpro/developer/ai-test-copilot-datasets/","by":"JOP-AL","license":"Copyright (c) Microsoft Corporation. CC BY 4.0, https://creativecommons.org/licenses/by/4.0/; code samples MIT. From https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/ai-test-copilot-datasets, MicrosoftDocs/dynamics365smb-devitpro-pb at 828b6ee (2026-11-19). Changed: includes filled in, links to this site, and for agents shortened without dropping a fact.","credit":"Microsoft Corporation, CC BY 4.0","origin":"https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/ai-test-copilot-datasets","source":{"name":"Business Central developer and IT pro docs","repo":"https://github.com/MicrosoftDocs/dynamics365smb-devitpro-pb","commit":"828b6ee72fcd75755627cd6011f3d20899217ab2","date":"2026-11-19"},"checked":"2026-11-19","description":"Learn how to create datasets for AI tests and agent tests in Business Central.","date":"2026-10-02","meta":{"topic":"concept-article"},"outline":[{"level":2,"title":"AI test datasets","slug":"ai-test-datasets","at":619,"chars":2021},{"level":3,"title":"Define a JSONL dataset","slug":"define-a-jsonl-dataset","at":940,"chars":746},{"level":3,"title":"Define a YAML dataset","slug":"define-a-yaml-dataset","at":1686,"chars":954},{"level":2,"title":"Agent test datasets","slug":"agent-test-datasets","at":2640,"chars":6449},{"level":3,"title":"File kinds","slug":"file-kinds","at":3093,"chars":1142},{"level":3,"title":"Top-level keys","slug":"top-level-keys","at":4235,"chars":618},{"level":3,"title":"Suite-setup file","slug":"suite-setup-file","at":4853,"chars":816},{"level":3,"title":"Test dataset file","slug":"test-dataset-file","at":5669,"chars":523},{"level":3,"title":"Query","slug":"query","at":6192,"chars":1077},{"level":3,"title":"Expected data","slug":"expected-data","at":7269,"chars":1028},{"level":3,"title":"Per-turn setup","slug":"per-turn-setup","at":8297,"chars":792},{"level":2,"title":"Placeholders for dates","slug":"placeholders-for-dates","at":9089,"chars":1320},{"level":2,"title":"Backwards compatibility: `question:`","slug":"backwards-compatibility-question","at":10409,"chars":815},{"level":2,"title":"Get data for your tests","slug":"get-data-for-your-tests","at":11224,"chars":1838},{"level":3,"title":"Sources of data for your tests","slug":"sources-of-data-for-your-tests","at":11572,"chars":985},{"level":3,"title":"More tips for collecting test data","slug":"more-tips-for-collecting-test-data","at":12557,"chars":505},{"level":2,"title":"Related information","slug":"related-information","at":13062,"chars":596}],"text":"A dataset is the foundation of AI testing in Business Central. Since AI tests are inherently data-driven, datasets allow you to simulate various user scenarios and interactions. By using diverse and comprehensive datasets, you can thoroughly evaluate AI features to ensure they meet high standards for correctness, safety, and accuracy.\nThis article covers two dataset flows:\n AI test datasets - simple input/expected-data pairs for evaluating prompt-based Copilot features. JSONL or YAML.\n Agent test datasets - richer YAML datasets that drive multi-turn agent scenarios with intervention handling and per-turn setup.\n## AI test datasets\nAI tests rely on datasets defined in either JSONL or YAML format. These datasets contain both the test input and the expected values used by Evaluation.\n> TIP\n> The full source code for the example used in this section is available in the Marketing Text Simple<ai-test-copilot-testtool> demo project.\n### Define a JSONL dataset\nWhile there's no rigid schema required, Evaluation supports certain common elements like test_setup and expected_data. Using these keywords helps create a consistent structure.\nHere's an example of a valid JSONL dataset:\n```json\n{\"test_setup\": {\"item_no\": \"C-10000\", \"description\": \"Contoso Coffee Machine\", \"uom\": \"PCS\"}, \"expected_data\": {\"tagline_max_length\": 20}}\n{\"test_setup\": {\"item_no\": \"C-10001\", \"description\": \"Contoso Toaster\", \"uom\": \"PCS\"}, \"expected_data\": {\"tagline_max_length\": 20}}\n{\"test_setup\": {\"item_no\": \"C-10002\", \"description\": \"Contoso Microwave Oven\", \"uom\": \"PCS\"}, \"expected_data\": {\"tagline_max_length\": 20}}\n```\nEach line represents a distinct test case with inputs and expected outputs.\n### Define a YAML dataset\nYou can also define the same dataset in YAML format for improved readability:\n```yaml\ntests:\n  - test_setup:\n      item_no: \"C-10000\"\n      description: \"Contoso Coffee Machine\"\n      uom: \"PCS\"\n    expected_data:\n      tagline_max_length: 20\n\n  - test_setup:\n      item_no: \"C-10001\"\n      description: \"Contoso Toaster\"\n      uom: \"PCS\"\n    expected_data:\n      tagline_max_length: 20\n\n  - test_setup:\n      item_no: \"C-10002\"\n      description: \"Contoso Microwave Oven\"\n      uom: \"PCS\"\n    expected_data:\n      tagline_max_length: 20\n```\nDatasets can also carry top-level metadata such as name and language, which is useful when you run multilingual evaluations:\n```yaml\nname: MARKETING-TEXT\ndescription: Marketing text generation accuracy tests.\nlanguage: en-US\ntests:\n  - test_setup:\n      item_no: \"C-10000\"\n      description: \"Contoso Coffee Machine\"\n      uom: \"PCS\"\n    expected_data:\n      tagline_max_length: 20\n```\n## Agent test datasets\nDatasets for agent tests describe what the agent receives on each turn and what the test should validate afterwards. They use a richer YAML schema with multi-turn support, intervention handling, and reusable suite-level setup.\n> TIP\n> A complete working example is available in the SalesValidationAgent3P<https://github.com/microsoft/BCTech/tree/master/samples/BCAgents/SalesValidationAgent/test> sample on the BCTech repository.\n### File kinds\nTwo YAML file kinds make up an agent test suite. The framework loads `*.yaml` recursively from the test app's `.resources/` folder, so folder names are convention.\nKind|Convention path|Top-level shape|Required\nSuite-setup file|`.resources/suite_setup/<NAME>.yaml`|`name:` + `suite_setup:` as object content|optional\nTest dataset file|`.resources/datasets/<NAME>.yaml`|`suite_setup:` as scalar reference + `tests:` array|required\nA suite-setup file holds data that needs to be set up only once across all tests in the suite - typical examples are master records every test relies on (locations, customers, posting groups). The framework runs it before the first test and skips it on subsequent turns; test code gates re-execution via `IsSuiteSetupDone()` and `SetEvalSuiteSetupCompleted()`.\nA test dataset file is the per-suite input that the test runner iterates over. It usually references a suite-setup by name but can stand alone if no shared setup is needed.\n> NOTE\n> Author one file per language for localized suites - for example, `MY-AGENT.yaml` and `MY-AGENT-FR.yaml`. The same convention applies to suite-setup files.\n### Top-level keys\nKey|Type|Required|Notes\nname|scalar|required on setup files|Group name. Other datasets reference a suite-setup by this exact string.\ndescription|scalar|optional|Free-form.\nlanguage|scalar|optional|Windows language tag (`en-US`, ...). Used to pick a language-variant suite-setup.\nsuite_setup|scalar or object|optional|Scalar value references another group's `name:`. Object value is inline content (used in suite-setup files).\ntests|array|required for test datasets|One element per test case.\ncontinue_on_failure|bool|optional|Per-test flag - when true, turn iteration continues after a failed turn.\n### Suite-setup file\nA suite-setup file declares its name (so dataset files can reference it) and provides inline `suite_setup:` content.\n```yaml\nname: MY-AGENT-SETUP\ndescription: Shared master data for the My Agent suite.\nsuite_setup:\n  setup_actions:\n    - action_type: create_location\n      action_data:\n        - Name: MAIN\n    - action_type: create_customers\n      action_data:\n        - \"No.\": CUST001\n          Name: Validation Customer 01\n        - \"No.\": CUST002\n          Name: Validation Customer 02\n```\nThe shape under suite_setup (setup_actions / action_type / action_data) is a convention defined by your test app - the framework imposes no schema. Your test library walks the JSON, dispatches on action_type, and creates records. See #per-turn-setup for the same convention applied at the turn level.\n### Test dataset file\nA test dataset file references a suite-setup (when used) and contains the `tests:` array.\n```yaml\nsuite_setup: MY-AGENT-SETUP\ntests:\n  - name: AGENT_TEST_01\n    description: Process a sales order with a future shipment date.\n    turns:\n      - turn_setup: { ... }     # see \"Per-turn setup\"\n        query: { ... }          # see \"Query\"\n        expected_data: { ... }  # see \"Expected data\"\n```\nAlways use the `turns:` array, even for single-turn tests. Multi-turn syntax is the only supported shape.\n### Query\n`query:` describes what the agent receives on a turn. It takes one of two shapes:\nTask input - used when starting a new agent task (typically turn 1, or any time the agent isn't already paused).\n```yaml\nquery:\n  from: Jane Doe                       # sender display name\n  title: \"Release all open sales orders\"\n  message: \"Release all open sales orders\"\n  attachments:                         # optional\n    - file: invoices/inv-001.pdf\n```\ntitle is required. from and message together add a task message; `attachments[].file` paths are resolved against the test app's resources.\nIntervention continuation - used when the agent paused on the previous turn and is waiting for the user.\n```yaml\nquery:\n  intervention:\n    suggestion: PROVIDE_DATE           # respond with a suggestion code offered by the agent\n    # OR\n    instruction: \"Use shipment date $DateFormula-<CW+1M>$\"   # respond with free-text\n```\nintervention.suggestion and intervention.instruction are mutually exclusive. Mixing both in one query, or mixing title and intervention, results in an error.\n### Expected data\n`expected_data:` is read by your validator. Most keys are application-defined and implemented in your test library. One sub-key is recognized by the framework: intervention_request.\n```yaml\nexpected_data:\n  intervention_request:                # framework-recognized\n    type: Assistance                   # see valid values below\n    suggestions:                       # optional list of codes that MUST be present\n      - PROVIDE_DATE\n  expected_released_count: 1           # application-defined; your validator reads this\n```\ntype is the English name of a value of the `Agent User Int Request Type` enum: Assistance, ReviewMessage, ReviewRecord, ResumeTask, Retry, Approval, or Permission. suggestions is optional - when present, every listed code must appear in the agent's actual intervention request.\nDeclare intervention_request on every turn where you expect the agent to pause. The framework fails the turn if the agent pauses without an expected intervention, or fails to pause when one is expected.\n### Per-turn setup\n`turn_setup:` holds data to materialize in Business Central before the agent runs. The shape is application-defined - most agents use the same setup_actions convention as the suite-setup, so one dispatcher in the test library can drive both.\n```yaml\nturn_setup:\n  setup_actions:\n    - action_type: create_sales_order\n      action_data:\n        - \"Sell-to Customer No.\": CUST001\n          \"Shipping Advice\": Complete\n          \"Shipment Date\": \"$DateFormula-<CW+1M>$\"\n          lines:\n            - Quantity: 10\n              \"Location Code\": MAIN\n```\nRecommendation: match AL field captions exactly (quoted when they contain spaces or special characters), and use placeholders for dates so the test doesn't drift over time. See the next section for the placeholder syntax.\n## Placeholders for dates\nHardcoded dates in datasets cause tests to need frequent maintenance as time passes - a test written for a January 2027 shipment date stops behaving the same way once January 2027 is in the past. Datasets express dates through placeholders that are calculated relative to WorkDate at run time.\nForm|Resolves to|Example\n`$DateFormula-<formula>$` (whole value)|Date|`\"$DateFormula-<CW+1M+1D>$\"`\n`$DateFormula-<formula>$` (inside text)|substring replaced with the formatted date|`\"Release orders for $DateFormula-<CW+1M>$\"`\n`$DateTimeFormula-<formula>$`|DateTime (time = `0T`)|`\"$DateTimeFormula-<CD>$\"`\n`$DateTimeFormula-<formula>-HH:MM:SS$`|DateTime with explicit time|`\"$DateTimeFormula-<CW>-13:30:11$\"`\n`$DateTimeFormula-<formula>-HH:MM:SS.FFFF$`|DateTime with milliseconds|`\"$DateTimeFormula-<CW>-13:30:11.1301$\"`\n`<formula>` is a standard AL DateFormula evaluated against WorkDate. Common values: `<CD>` (today), `<CW>` (start of current week), `<CM>` (start of current month), `<CW+1M+1D>`, `<-7D>`, `<+30D>`.\n> IMPORTANT\n> Always quote placeholder strings in YAML - the `<` and `>` characters are otherwise interpreted as flow-style delimiters by some parsers.\nResolution is automatic when YAML values are read through the test framework. Authors don't need to call a resolver explicitly.\n## Backwards compatibility: `question:`\nEarlier agent test datasets used `question:` instead of `query:`. The framework still accepts the legacy form so existing tests don't have to be rewritten, but new datasets should use `query:`:\nLegacy|Current\n`question:` (scalar text or object with from / message)|`query:` with title / from / message / attachments, or query.intervention for continuations\n`AITTestContext.GetQuestion()`|`AITTestContext.GetQuery()`\nThe current `query:` shape is required to use multi-turn continuation queries (query.intervention.suggestion / `.instruction`), the data-driven driver LibraryAgent.RunTurnAndWait, and the framework's automatic intervention validation. See Write agent tests<ai-test-copilot-agent-tests#backwards-compatibility-question-and-getquestion> for migration guidance.\n## Get data for your tests\nWhen you create AI tests, the data you use is as important as the AI features you're testing. Quality, consistency, and realism of data are critical for ensuring that your tests are comprehensive and meaningful.\n> TIP\n> See Best practices<ai-test-copilot-bestpractices> about other considerations when creating datasets.\n### Sources of data for your tests\n1. Public datasets:\n  There are many publicly available datasets that you can use for AI testing.\n2. Synthetic data:\n  In cases where real-world data is difficult to obtain or too sensitive, you can generate synthetic data.\n  Synthetic data can be especially useful for testing edge cases or generating large volumes of data quickly.\n3. Customer or internal data:\n  If you have access to anonymized customer data or internal business datasets, this can be a valuable source for realistic AI testing.\n  Ensure that the data is appropriately anonymized and that you comply with privacy regulations.\n4. Crowdsourced data:\n  Certain platforms allow you to gather custom data by using crowdsourcing.\n5. Simulated data from domain experts:\n  In certain domains, domain experts can provide valuable insights into generating realistic and relevant test data.\n  This approach is helpful when real-world data isn't readily available or too sensitive to share.\n### More tips for collecting test data\n Start with small datasets: Especially when testing new AI features, begin with small, manageable datasets to avoid overwhelming your testing process.\n Incrementally increase complexity: As you refine your tests, increase the complexity of the datasets to better simulate real-world scenarios.\n Document data sources: Always document the origins of your test data, including any transformations made, to ensure traceability and transparency in your testing process.\n## Related information\nBusiness Central Copilot Evaluation<https://github.com/microsoft/BCApps/blob/main/src/Tools/AI%20Test%20Toolkit/README.md>\nSales Validation Agent sample on BCTech<https://github.com/microsoft/BCTech/tree/master/samples/BCAgents/SalesValidationAgent/test>\nBuild the Copilot capability in AL<ai-build-capability-in-al>\nTest the Copilot capability in AL<ai-test-copilot>\nEvaluation<ai-test-copilot-testtool>\nWrite AI tests<ai-test-copilot-ai-tests>\nWrite agent tests<ai-test-copilot-agent-tests>\nBest practices for testing the Copilot capability<ai-test-copilot-bestpractices>"}