Created
Jun 22, 2024 03:04 AM
Favorite
Favorite
Priority
备注
推荐
🌟🌟🌟🌟
类型
模型测试
note
Contact us if you're dealing with sensitive data that has to reside in your private VPCs.
Create Your First Test Case
Run
touch test_example.py
to create a test file in your root directory. Open test_example.py
and paste in your first test case:test_example.py
Run
deepeval test run
from the root directory of your project:Congratulations! Your test case should have passed ✅ Let's breakdown what happened.
- The variable
input
mimics a user input, andactual_output
is a placeholder for what your application's supposed to output based on this input.
- The variable
retrieval_context
contains the retrieved context from your knowledge base, andAnswerRelevancyMetric(threshold=0.5)
is an default metric provided bydeepeval
for you to evaluate your LLM output's relevancy based on the provided retrieval context.
- All metric scores range from 0 - 1, which the
threshold=0.5
threshold ultimately determines if your test have passed or not.
tip
You'll need to set your
OPENAI_API_KEY
as an enviornment variable before running the AnswerRelevancyMetric
, since the AnswerRelevancyMetric
is an LLM-evaluated metric.To use ANY custom LLM of your choice, check out this part of the docs.
You can also save test results locally for each test run. Simply set the
DEEPEVAL_RESULTS_FOLDER
environment variable to your relative path of choice:Create Your First Metric
deepeval
provides two types of LLM evaluation metrics to evaluate LLM outputs: plug-and-use default metrics, and custom metrics for any evaluation criteria.Default Metrics
deepeval
offers 14+ research backed default metrics covering a wide range of use-cases (such as RAG and fine-tuning). To create a metric, simply import from the deepeval.metrics
module:Note that you can run a metric as a standalone or as part of a test run as shown in previous sections.
info
All default metrics are evaluated using LLMs, and you can use ANY LLM of your choice. For more information, visit the metrics introduction section.
Custom Metrics
deepeval
provides G-Eval, a state-of-the-art LLM evaluation framework for anyone to create a custom LLM-evaluated metric using natural language. Here's an example:Under the hood,
deepeval
first generates a series of evaluation steps, before using these steps in conjuction with information in an LLMTestCase
for evaluation. For more information, visit the G-Eval documentation page.tip
If you wish to customize your metrics a bit more, you can choose to implement your own metric. You can find a comprehensive step-by-step guide here, but here's a quick example of how you can create a metric that is NOT evaluated using LLMs:
You'll notice that although not documented,
deepeval
additionally offers a scorer
module for more traditional NLP scoring method and can be found here.You can also create a custom metric to combine several different metrics into one. For example. combining the
AnswerRelevancyMetric
and FaithfulnessMetric
to test whether an LLM output is both relevant and faithful (ie. not hallucinating).Measure Multiple Metrics At Once
To avoid redundant code,
deepeval
offers an easy way to apply as many metrics as you wish for a single test case.test_example.py
In this scenario,
test_everything
only passes if all metrics are passing. Run deepeval test run
again to see the results:info
deepeval
optimizes evaluation speed by running all metrics for each test case concurrently.Create Your First Dataset
A dataset in
deepeval
, or more specifically an evaluation dataset, is simply a collection of LLMTestCases
and/or Goldens
.note
A
Golden
is simply an LLMTestCase
with no actual_output
, and it is an important concept if you're looking to generate LLM outputs at evlauation time. To learn more about Golden
s, click here.To create a dataset, simply initialize an
EvaluationDataset
with a list of LLMTestCase
s or Golden
s:Then, using
deepeval
's Pytest integration, you can utilize the @pytest.mark.parametrize
decorator to loop through and evaluate your dataset.test_dataset.py
tip
You can also evaluate entire datasets without going through the CLI (if you're in a notebook environment):
Additionally, you can avoid re-evaluating test cases by reading from
deepeval
's local cache using the optional -c
flag:Or run test cases in parallel by using the optional
-n
flag followed by a number (that determines the number of processes that will be used) when executing deepeval test run
:Visit the evaluation introduction section to learn about the different types of flags you can use with the
deepeval test run
command.Generate Synthetic Datasets
deepeval
offers a synthetic data generator that uses state-of-the-art evolution techniques to make synthetic (aka. AI generated) datasets realistic. This is especially helpful if you don't have a prepared evaluation dataset.caution
You should aim to manually inspect and edit any synthetic data where possible.
Simply supply a list of local document paths to generate a synthetic dataset from your knowledge base.
Note that
deepeval
's synthesizer does NOT generate actual_output
s for each golden. This is because actual_output
s are meant to be generated by your LLM (application), not deepeval
's synthesizer.Visit the synthesizer section to learn everything about how
deepeval
's synthesizer works.note
Remember, a
Golden
is basically an LLMTestCase
but with no actual_output
.Using Confident AI
If you have reached this point, you've likely ran
deepeval test run
multiple times. To keep track of all future evaluation results created by deepeval
, login to Confident AI by running the following command in the CLI:Confident AI is the platform that unlocks
deepeval
's full potential, and allows anyone to easily:- keep track of and debug historical test run results
- discover optimal hyperparameters, such as the best models and prompt templates to use
- centralize and synthesize evaluation datasets on the cloud
- safeguard against breaking changes in CI/CD pipelines
- run real-time evaluations in production, with custom metrics
info
Click here for the full documentation on using Confident AI with
deepeval
.Follow the instructions displayed on the CLI to create an account, get your Confident API key, and paste it in the CLI. You should see a message congratulating your successful login.
Once logged in, you'll be able to view test run results on Confident AI each time you execute a test run:
test_example.py
You should now see a link being returned upon test completion. Paste it in your browser to view results.
Managing Datasets On The Cloud
deepeval
allows you to push and pull datasets stored on Confident AI. This is similar to pushing and pulling a repo from GitHub. To push a dataset to Confident AI, create an EvaluationDataset
instance, populate it with test cases, and use the push()
method:You can now edit, comment on, and manage test cases on the cloud instead of locally in a CSV file.
info
You can also create a dataset on Confident AI without going through
deepeval
.To pull the dataset for evaluation, use the
pull()
method and evaluate it as shown in previous sections:test_example.py
tip
In reality, you'll often times want to process the pulled dataset before evaluating it, since test cases in a dataset are stored as
Golden
s, which might not always be ready for evaluation (ie. missing an actual_output
).To see a concrete example and a more detailed explanation, visit the evaluating datasets section.
Optimizing Hyperparameters
Confident AI helps you easily discover the optimal set of hyperparameters, which in
deepeval
refers to properties such as the models, prompt templates, etc. used when generating the actual_output
s for each LLMTestCase
.To discover which set of hyperparameters gives you the best evaluation metrics results, use the
@deepeval.log_hyperparameters
decorator:test_example.py
note
The
hyperparameters()
function DOESN'T necessarily have to be named 'hyperparameters'. All you need in order to log hyperparameters on Confident AI is to decorate a function that returns a valid dictionary.Once you've added this decorator, execute
test_example.py
once more:The
@deepeval.log_hyperparameters
decorator helps Confident AI keep track of the hyperparameters used when generating the actual_output
s for a particular test run. This allows you to identify which combination of hyperparameters gives the best evaluation metric results over time.tip
You can also log hyperparameters via the
evaluate()
function:Feel free to execute this in a nested for loop to figure out the best hyperparameter combination!
Monitoring LLM Responses
Confident AI allows anyone to monitor and evaluate LLM responses in real-time. A single API request is all that's required, and this would typically happen at your servers right before returning an LLM response to your users:
Confident AI will automatically run evaluations for each incoming LLM response on metrics you have turned on. Simply head over to the 'Project Details' page on Confident AI to turn on these real-time metrics.
info
You can find more information on running real-time evaluations here.
Sending Human Feedback
Confident AI allows you to send human feedback on LLM responses tracked in production, all via one API call by using the previously returned
event_id
from deepeval.track()
:Confident AI allows you to keep track of either
"user"
feedback (ie. feedback provided by end users interacting with your LLM application), or "reviewer"
feedback (ie. feedback provided by reviewers manually checking the quality of LLM responses in production).note
To learn more, visit the human feedback section page.
Full Example
You can find the full example here on our Github.
by