Contact

Testing AI Solutions

At its basis, testing AI applications is not fundamentally different when compared to “standard” applications; the same key principles apply to them as well: discover defects, testing should be performed on a known version of a piece of software, ensure effective coverage, test early, prioritize, implement quality gates and foster a quality culture in the project team.

Learn More

Fundamental Principles

When it comes to the implementation of these fundamental principles, there are significant differences that can make testing of GenAI apps a daring challenge. GenAI applications that leverage large language models (LLMs) require a different approach. Testing the AI model is most times a black box task that requires expertise in black box testing, with the ability to replicate the same test hundreds/thousands of times and use non-deterministic assertions/validations. 

Testing text-based GenAI apps is testing a combination of the RAG contextual data, the LLM itself, Prompts, Model Temperature and other settings.

Testing Challenges

Typical challenges in testing GenAI apps are:

  • Drifting. The changing nature of LLMs, coupled with the difficulties of nailing down a specific and known version of the test object means that the behavior of the LLM will change over time, a phenomenon known as drift. Drift can mean that the same questions posed to a model start to change fractionally over time.   These changes can include the introduction of bias and can, if left unchecked, result in models that show significant divergence from what was originally viewed as acceptable.
  • Input and/or Output is unstructured or non-text, like image, audio, or video (multi-model).
  • Many GenAI applications are interactive/chat-alike, which requires multiple interaction steps/flows for a single test case, adding complexity.
  • Debugging. Investigating the source of identified problems in a GenAI application is significantly more difficult than in a traditional software application.

Testing generative AI applications requires creative thinking and the use of specific strategies and techniques. Working with a partner having rich experience in testing GenAI applications will make a key difference.

Type of AI Testing

Some common types of AI testing that we perform at Apsisware are:

  • Input Data quality assessment/testing. AI solutions are heavily reliant on the quality of data used for input (documents, and other structured and unstructured data). Testing is required to ensure that the input data is accurate, reliable, and representative.
  • Functional testing, which includes positive and negative scenarios, accuracy testing, completeness testing, bias and toxicity testing/adversarial testing, and hallucination testing. We identify a series of questions that are based on the expected use of the GenAI system; these will be questions that have known, factual answers, that can be easily assessed. The questions and answers can be used by an automation tool or test harness, they should be saved in a csv file or database and attached with a date and a version number. We define a series of scoring levels for accuracy and completeness, that form a benchmark truth and can be used in testing to monitor drift and to ensure the bias and fairness is at the desired values.
  • Non-functional testing: We perform Performance testing, Scalability testing, Internationalisation and Localization testing, Security testing, Documentation testing, Legal Compliance testing, and Accessibility testing.

Whenever possible, we try to automate testing  and add the test automation suite into a CI/CD tool so they can be run automatically, to lower the project manual testing costs.

AI Testing Tools

Besides the common testing tools and frameworks, we use a variety of specific AI testing tools. We believe in choosing the right tools for the job, to address the challenges of the specific project. Here is a list of such AI testing tools, that can also get you inspired:

Evaluate and monitor for model health, accuracy, drift, bias and quality

  • Watsonx.governance by IBM
  • Azure AI Studio

Debug, collaborate, test, and monitor your LLM applications:

  • https://www.langchain.com/langsmith | Langsmith by Langchain
  • https://www.giskard.ai/ | Giskard
  • https://www.promptfoo.dev/ | Promptfoo
  • https://python.langchain.com/v0.1/docs/guides/productionization/evaluation/ | LangChain evaluators

Evaluation framework for your RAG pipelines

  • https://github.com/explodinggradients/ragas | ragas

Evaluate AI models

  • https://github.com/braintrustdata/autoevals | AutoEvals
  • https://github.com/confident-ai/deepeval | DeepEval
  • https://github.com/rungalileo | Galileo

Optimise LM

  • https://github.com/stanfordnlp/dspy | DSPy

Performance testing

  • https://github.com/ray-project/llmperf | LLMPerf
  • https://benchllm.com/ | BenchLLM

Prompt tools/engineering

  • https://github.com/hegelai/prompttools/ | Prompttools
  • https://chainforge.ai/ | ChainForge

Protect against LLM vulnerabilities

  • https://github.com/NVIDIA/NeMo-Guardrails | Nvidia NeMo-Guardrails
  • https://www.guardrails.io/ | Guardrails

Test GenAI applications using a LLM:

  • https://github.com/AtomicJar/genai-test | https://github.com/AtomicJar/genai-test

We use AI Tools for Testing AI Solutions

Not the least, we use the AI/GenAI capabilities for testing, to increase test speed and efficiency. GenAI is a great solution to automate repetitive tasks and generate data-driven insights. AI can automatically generate test cases using machine learning algorithms, reducing manual test design effort. Tests can be executed automatically using AI-powered tools. AI testing tools can analyse large sets of data and identify patterns and anomalies fast, without human intervention.

Our QA team analyses the project and proposes the most efficient approach for testing. Quality is a must-do, never to compromise key value for us. <Contact us> to find out more about us and how we work.

Contact Us
Top