Experiment on LLM prompts and AI-agent configurations

LLM-powered applications and AI agents are non-deterministic. Changing a prompt, model, retrieval strategy, tool configuration, or agent workflow can affect response quality, operating cost, latency, and user behavior in different, hard-to-predict ways. Evaluation frameworks such as RAGAS score whether an individual response meets a quality bar, but that score doesn’t tell you whether the change helps users accomplish what they came to your product to do. You can use Kameleoon Feature Experimentation capabilities to customize, test, and roll out the configuration behind a generative AI application or AI agent. A feature variable holds a single piece of that configuration (a prompt, a model parameter, a retrieval strategy, a tool definition), so your team can manage it outside your application code. Each variation represents a candidate configuration, which lets you iterate, experiment, and release changes more safely, without redeploying. Split traffic between variations and compare their impact using several types of metrics:

• AI-quality metrics, such as correctness, groundedness, relevance, safety, or compliance
• Agent-performance metrics, such as successful tool use or task completion
• Operational metrics, such as latency, token consumption, errors, and cost
• User and business metrics, such as satisfaction, escalation, conversion, retention, and revenue

Because the configuration lives in a feature flag instead of your source code, you can add, edit, or roll back a variation directly from the Kameleoon platform at any time.

​

Evaluation and experimentation solve different problems

Evaluation determines whether an individual model or agent output meets a defined quality standard. Observability tools let you inspect the prompts, responses, traces, retrieval steps, and tool calls behind that output. Experimentation determines whether a change to the underlying configuration causes a measurable improvement for users or the business: a question neither evaluation nor observability answers. For example, an LLM judge might score one prompt as more grounded than another. A Kameleoon experiment tells you whether that same prompt also improves task completion, reduces escalation, increases satisfaction, or affects latency and cost, the outcomes your team is accountable for. Kameleoon doesn’t replace your LLM observability or evaluation stack. Feed evaluator scores from RAGAS, an LLM judge, or a human review process into Kameleoon as a custom goal with a numeric value, and track them alongside the behavioral and business goals your experiment already measures, connecting model-level quality signals to statistically reliable measurement of real user impact. For most AI experiments, combine several metric types rather than relying on one:

• Set a user or business outcome as the primary goal.
• Track AI-quality metrics as secondary goals or guardrails.
• Monitor latency, cost, errors, and safety as operational guardrails.
• Validate automated judges against a sample of human-reviewed examples before you trust their scores at scale.

​

How it works

A feature flag stores each configuration (such as a prompt) as the value of a feature variable, with one variation per option you want to test. When a visitor reaches your application, the Kameleoon SDK assigns the visitor to a variation and returns the corresponding value, which your application code uses to call the LLM or configure the agent. A goal attached to the feature flag records a conversion when the visitor interacts with your LLM-powered feature or AI agent, such as asking a follow-up question or completing a task with its help. After you collect enough traffic, compare each variation’s conversion rate, along with any AI-quality or operational metrics you’re tracking, to decide which configuration performs best.

​

Prerequisites

• A Kameleoon account with a project set up for feature experimentation.
• Your account’s client ID and client secret. To find these values, see API credentials.
• A Python application where you can install the Kameleoon SDK.

​

Set up your prompt experiment in Kameleoon

Configure the feature flag, prompt variations, and tracking goal in the Kameleoon platform before you touch your application code.

​

Create the feature flag

Create a feature flag to hold your prompt variations and control the rollout of your experiment.

1. In the Kameleoon app, click Features > Flags & Experiments > New feature flag.
2. Enter a name, for example LLM prompt test, and select the project for the flag.
3. In the Description field, note what the flag controls, so other members of your team understand its purpose.
4. Click Validate.

For more detail, see Create a feature flag.

​

Store the prompt in a feature variable

Add a feature variable to hold the prompt text, so you can change it from the Kameleoon platform without editing your application code.

1. On the flag’s page, in the left sidebar, click Set Up > Variables > Add Variable.
2. Set the variable’s Type to String.
3. Enter a Variable Key, for example prompt_template.
4. Set the Default Value to the prompt your application currently uses. Kameleoon delivers this value to visitors who aren’t part of your experiment.
5. Click Save.

For more detail, see Define feature variables.

​

Create a variation for each prompt

Create one variation per prompt you want to test, and set the prompt_template variable to the corresponding text in each.

1. In the left sidebar, click Set Up > Variations > Add variation.
2. Enter a Name for the variation, for example Detailed summary.
3. Set the prompt_template variable to the prompt text for this variation.
4. Click Save.
5. Repeat these steps for each additional prompt you want to test.

For more detail, see Define feature variations.

​

Attach a goal to measure engagement

Attach a goal to the feature flag so Kameleoon can measure which prompt variation drives more engagement. Because Kameleoon is a unified platform, you can attach any goal that already exists in your organization, such as a transaction goal defined by another team, or create a goal specific to your LLM-powered feature.

1. On the flag’s page, in the Set Up menu, click Goals > Add goal.
2. Select an existing goal, or click Create a new goal to define one, such as a custom goal that fires when a visitor interacts with your LLM-powered feature.
3. Click Save.

For more detail on goal types, including how to trigger a custom goal from your backend, see Create a goal.

​

Roll out the experiment

Create an experiment rule that splits traffic between your prompt variations, then turn on the environment to start collecting data.

1. In the Rollout Planner, select the environment you want to target, for example Production.
2. Click Add a rule > Experiment.
3. Under Variations to serve, add each prompt variation and set its exposition percentage. For example, split traffic evenly between two variations at 50% each.
4. Set the rule’s targeting to include the visitors you want to test, for example all visitors reaching the application.
5. Turn the environment’s ON/OFF toggle to ON.
6. Click Save.

For more detail, see Create feature experiments. Once you save the rule, Kameleoon starts assigning visitors to a variation and serving the corresponding prompt. To change a prompt or add a variation later, edit it directly in the Kameleoon platform. You don’t need to redeploy your application to make these changes.

​

Retrieve the prompt in your application

Install the Kameleoon Python SDK, then retrieve the visitor’s assigned prompt and track a conversion when the visitor interacts with your LLM-powered feature. The same pattern applies to any Kameleoon server-side SDK, including Node.js, Java, and Go.

1. Install the SDK as a dependency:
   pip install kameleoon-client-python
   
2. Initialize the client with your site code and credentials:
   from kameleoon import KameleoonClient, KameleoonClientConfig, KameleoonClientFactory
   
   SITE_CODE = "a8st4f59bj"
   FEATURE_KEY = "llm_prompt_test"
   GOAL_ID = 12345  # Replace with the ID of the goal you attached to the flag
   
   configuration = KameleoonClientConfig(
       client_id="your-client-id",
       client_secret="your-client-secret",
   )
   kameleoon_client = KameleoonClientFactory.create(SITE_CODE, configuration)
   await kameleoon_client.wait_init_async()
   
3. Retrieve the assigned prompt before you call your LLM, and track a conversion when the visitor interacts with the LLM-powered feature:
   def get_prompt_for_visitor(visitor_code: str) -> str:
       variation = kameleoon_client.get_variation(visitor_code, FEATURE_KEY)
       return variation.variables["prompt_template"].value
   
   
   def track_llm_interaction(visitor_code: str) -> None:
       kameleoon_client.track_conversion(visitor_code, GOAL_ID)
   
   Call get_prompt_for_visitor() with the visitor’s visitor_code before you send a request to your LLM, and use the returned value as the prompt. Call track_llm_interaction() when the visitor interacts with the LLM-powered feature, such as submitting a question or receiving a response.

​

Monitor and iterate

Open the feature flag’s results page to compare the conversion rate of each prompt variation against the goal you attached. Kameleoon tracks exposures and conversions automatically whenever your application calls get_variation() and track_conversion(), so you don’t need any additional instrumentation. If you also attached an AI-quality or operational metric as a goal, compare it alongside conversion rate before you decide which variation to roll out. For more detail, see Analyze a feature flag’s overall results.

​

Next steps

• Read the Python SDK reference for advanced options such as custom data, cross-device experimentation, and targeting conditions.
• Attach precise segmentation criteria to target the experiment at a specific audience.
• Explore feature variables to variabilize other parts of your LLM-powered feature or AI agent, such as model parameters, retrieval settings, or tool definitions.
• Attach multiple goals to the same flag, such as an AI-quality score from your evaluation pipeline alongside a business metric, to compare configurations across several dimensions at once. See Create goals for feature flags.