E2E Reference

================================

Quick Start Slides

Check out the Quick Start Slides for an overview of DaC and how to get started. It provides some example CLI commands using the detection-rules repo.

If you’re starting from scratch and would like to get quickly started, here are the high-level instructions. These steps assume you are familiar with the considerations associated with the various steps and elect to follow the detection-rules VCS approach to manage rules. It also assumes advanced configurations are not applied.

Demo Video

Take a look at an example of how you can use some of our DaC features. The repo used in this example can be found here

Quick Start Example Detection Rules CLI Commands

For the full walkthrough (setup, Elastic connection, rules, unit tests, schemas), see the DaC Quick Start Guide.

Steps:

Clone the detection rules repo and install python requirements, see getting started.
Initialize a custom rules directory via the setup config command.

python -m detection_rules custom-rules setup-config dac_custom_rules_dir
Edit the _config.yaml file in your dac_custom_rules_dir directory.

Most users will want to add these additional parameters for ease of use:
```
bypass_version_lock: True
normalize_kql_keywords: True
auto_gen_schema_file: "etc/schemas/auto_gen.json"
bypass_optional_elastic_validation: True
```
For finer control (for example bypassing only local ES|QL keep or METADATA checks), use the individual bypass_* keys in _config.yaml documented in the DaC Quick Start Guide (section 6. Custom configuration (_config.yaml)) and in upstream detection_rules/etc/_config.yaml; see detection-rules PR #5869.

Edit the test_config.yaml in your dac_custom_rules_dir/etc directory.

For unit testing, most users will also want to bypass the following Elastic unit tests.

unit_tests:
    bypass:
    - tests.test_gh_workflows.TestWorkflows.test_matrix_to_lock_version_defaults
    - tests.test_schemas.TestVersionLockSchema.test_version_lock_has_nested_previous
    - tests.test_packages.TestRegistryPackage.test_registry_package_config
    - tests.test_all_rules.TestValidRules.test_schema_and_dupes
    - tests.test_all_rules.TestRuleMetadata.test_invalid_queries
    - tests.test_all_rules.TestValidRules.test_bbr_validation
    - tests.test_all_rules.TestValidRules.test_rule_type_changes
    - tests.test_schemas.TestSchemas.test_eql_validation

Set your environment variable to use the custom configuration and rules directory you just made.

export CUSTOM_RULES_DIR="<full_path_to_dac_custom_rules_dir>"
If running locally, set your .detection-rules-cfg.json in the root of the detection rules directory. See setup a config file for more details.
```
{
    "cloud_id": "example:example",
    "api_key": "example",
}
```

Export rules and related exceptions to a Kibana Instance, overwriting existing, stripping version, and skipping errors.

python -m detection_rules kibana export-rules \
--skip-errors \
--strip-version \
--export-exceptions \
--export-action-connectors \
--directory my-custom-rules

Run local unit tests.

make test
Import rules and related exceptions to a Kibana Instance, overwriting existing.

💡 Note: Please remember to backup your rules! If you have not run kibana export-rules or used the UI to perform a bulk export of rules before running this command, please do so! Once rules are overwritten in Elastic Security/Kibana it may be impossible to recover them! See the FAQ for more info on backing up rules.
```
python -m detection_rules kibana import-rules \
--overwrite \
--overwrite-exceptions \
--overwrite-action-connectors
```
Or if you prefer to import and export using ndjson file(s) instead of the Kibana API use the following:

For moving rules from a Kibana rules export to the repo
```
python -m detection_rules import-rules-to-repo <ndjson_file> \
--required-only \
--action-connector-import \
--exceptions-import \
--default-author DefaultAuthor \
--skip-errors
```
For moving rules from the repo to an ndjson file that is compatible with Kibana rule import.
```
python -m detection_rules export-rules-from-repo \
--directory my-custom-rules \
--outfile my-custom-rules-export.ndjson \
--include-action-connectors \
--include-exceptions
```
For kibana export-rules, --directory is the output folder; for export-rules-from-repo, --directory loads rules from disk (input). To export YAML instead of a single NDJSON file, use --save-yaml-dir (and omit --outfile for that mode). To pull from Kibana as YAML instead of TOML, add --save-as-yaml to kibana export-rules. See Quick Start — Syncing with Kibana.

For more information on these CLI commands please see CLI.md, docs-dev/custom-rules.md, and docs-dev/detections-as-code.md in Detection Rules.

Locally

Steps:

Fork the detection-rules repo
Configure Elastic Security authentication
Install the python requirements
Initialize the custom rules directory (optionally configure schema and versioning files)
Configure unit tests (optionally bypass specific test cases)
Create a detection rule (optionally create exception and action list)
Validate the rules (optionally test queries in development environment)
Open pull request to store in VCS and peer review
Version lock your rules
Push merged rules to production environment (optionally version lock and sync via CICD)

In VCS

Steps:

Setup branch protections and code owners
Setup environment secrets and variables that will be used in CICD
Setup CICD workflows to validate and sync rules