Skip to content

Commit

Permalink
Doc: Add structure and content for air-gapped architectures (#654) (#804
Browse files Browse the repository at this point in the history
)

(cherry picked from commit 1f245d7)
Co-authored-by: Karen Metts <[email protected]>
  • Loading branch information
mergify[bot] committed Jan 11, 2024
1 parent 57d0c60 commit 6956349
Show file tree
Hide file tree
Showing 7 changed files with 92 additions and 23 deletions.
8 changes: 0 additions & 8 deletions docs/en/ingest-arch/20-airgap.asciidoc

This file was deleted.

15 changes: 15 additions & 0 deletions docs/en/ingest-arch/99-airgapped.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[[airgapped-env]]
== Elastic air-gapped architectures

You can deploy the {stack} with some or all components in a data center or other environment with no access to any outside networks.
Some stack components require additional configuration and local dependencies when deployed in air-gapped environments.

[discrete]
[[airgapped-archs]]
=== {agent} to {es} architectures: air-gapped environments

* <<agent-es-airgapped>>
* <<agent-ls-airgapped>>

include::99a-airgapped-ea-es.asciidoc[]
include::99b-airgapped-ea-ls.asciidoc[]
21 changes: 21 additions & 0 deletions docs/en/ingest-arch/99a-airgapped-ea-es.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
[[agent-es-airgapped]]
=== {agent} to {es}: Air-gapped environment

image::images/ea-es-airgapped.png[Image showing {agent} and {es} in an air-gapped environment]

Ingest model::
All {stack} components deployed inside a DMZ:
* Control path: {agent} to {fleet} to {es} +
* Data path: {agent} to {es}

Use when::
Your self-managed {stack} deployment has no access to outside networks

[discrete]
[[airgapped-es-resources]]
=== Resources

Info for air-gapped environments:

* {stack-ref}/air-gapped-install.html[Installing the {stack} in an air-gapped environment]
* {fleet-guide}/fleet-agent-proxy-support.html[Using a proxy server with Elastic Agent and Fleet]
30 changes: 30 additions & 0 deletions docs/en/ingest-arch/99b-airgapped-ea-ls.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
[[agent-ls-airgapped]]
=== {agent} to {ls}: Air-gapped environment

image::images/ea-ls-airgapped.png[Image showing {agent}, {ls}, and {es} in an air-gapped environment]

Ingest model::
All {stack} components deployed inside a DMZ:
* Control path: {agent} to {fleet} to {es} +
* Data path: {agent} to {es}

Use when::
Your self-managed {stack} deployment has no access to outside networks

[discrete]
[[airgapped-ls-resources]]
=== Resources

Info for air-gapped environments:

* {stack-ref}/air-gapped-install.html[Installing the {stack} in an air-gapped environment]
* {fleet-guide}/fleet-agent-proxy-support.html[Using a proxy server with Elastic Agent and Fleet]

[discrete]
[[ls-geoip]]
=== Geoip database management in air-gapped environments

The {logstash-ref}/plugins-filters-geoip.html[{ls} geoip filter] requires regular database updates to remain up-to-date with the latest information.
If you are using the {ls} geoip filter plugin in an air-gapped environment,
you can manage updates through a proxy, a custom endpoint, or manually.
Check out {logstash-ref}/plugins-filters-geoip.html#plugins-filters-geoip-manage_update[Manage your own database updates] for more info.
38 changes: 24 additions & 14 deletions docs/en/ingest-arch/arch-list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -31,49 +31,59 @@ Eventually, we may move original here, but for now it needs to stay in cloud inf
image:images/ea-es.png[Image showing {agent} collecting data and sending to {es}]

a|
* An {integrations-docs}[{agent} integration] is available for your data source:
** Software components with <<agent-installed,{agent} installed>>
** Software components using <<agent-apis,APIs for data collection>>
An {integrations-docs}[{agent} integration] is available for your data source:

* Software components with <<agent-installed,{agent} installed>>
* Software components using <<agent-apis,APIs for data collection>>


| <<agent-ls>>

image:images/ea-ls-es.png[Image showing {agent} to {ls} to {es}]

a|
* You need additional capabilities from {ls}:
** <<ls-enrich,*enrichment*>> between {agent} and {es}
** <<lspq,*persistent queue (PQ) buffering*>> to accommodate network issues and downstream unavailability
** <<ls-networkbridge,*proxying*>> in cases where {agent}s have network restrictions for connecting outside of the {agent} network
** data needs to be <<ls-multi,*routed to multiple*>> {es} clusters and other destinations depending on the content
You need additional capabilities offered by {ls}:

* <<ls-enrich,*enrichment*>> between {agent} and {es}
* <<lspq,*persistent queue (PQ) buffering*>> to accommodate network issues and downstream unavailability
* <<ls-networkbridge,*proxying*>> in cases where {agent}s have network restrictions for connecting outside of the {agent} network
* data needs to be <<ls-multi,*routed to multiple*>> {es} clusters and other destinations depending on the content


| <<agent-proxy>>

image:images/ea-proxy-es.png[Image showing connections between {agent} and {es} using a proxy]

a|
* Agents have <<agent-proxy,network restrictions>> that prevent connecting outside of the {agent} network
* Note that <<ls-networkbridge,{ls} as proxy>> is one option
Agents have <<agent-proxy,network restrictions>> that prevent connecting outside of the {agent} network
Note that <<ls-networkbridge,{ls} as proxy>> is one option.


| <<agent-kafka-es>>

image:images/ea-kafka.png[Image showing {agent} collecting data and using Kafka as a message queue enroute to {es}]

a|
* Kafka is your <<agent-kafka-es,middleware message queue>>:
** <<agent-kafka-essink,Kafka ES sink connector>> to write from Kafka to {es}
** <<agent-kafka-ls,{ls} to read from Kafka and route to {es}>>
Kafka is your <<agent-kafka-es,middleware message queue>>:

* <<agent-kafka-essink,Kafka ES sink connector>> to write from Kafka to {es}
* <<agent-kafka-ls,{ls} to read from Kafka and route to {es}>>


| <<ls-for-input>>

image:images/ls-es.png[Image showing {ls} collecting data and sending to {es}]

a|
* You need to collect data from a source that {agent} can't read (such as databases, AWS Kinesis).
You need to collect data from a source that {agent} can't read (such as databases, AWS Kinesis).
Check out the {logstash-ref}/input-plugins.html[{ls} input plugins].


| <<airgapped-env>>

image:images/ea-airgapped.png[Image showing {stack} in an air-gapped environment]

a|
You want to deploy {agent} and {stack} in an air-gapped environment (no access to outside networks)

|===
Binary file added docs/en/ingest-arch/images/ea-airgapped.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion docs/en/ingest-arch/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[]
include::{docs-root}/shared/attributes.asciidoc[]

[[ingest-architectures]]
= {es} Ingest Architectures
= Elastic Ingest Reference Architectures

include::arch-list.asciidoc[]

Expand All @@ -16,6 +16,7 @@ include::16-agent-kafka.asciidoc[]

include::8-ls-input.asciidoc[]

include::99-airgapped.asciidoc[]

// === Next set of architectures
// include::3-schemamod.asciidoc[]
Expand Down

0 comments on commit 6956349

Please sign in to comment.