Managing logging indices

You can use Elasticsearch Index Lifecycle Management features to define the retention period of your MI logging data.

Time series MI log data keeps accumulating over time. You can automate the management of this data by configuring a retention period which defines the minimum time the data will be stored in Elasticsearch. Data older than the retention period will be deleted automatically by Elasticsearch at a later time.

  1. In Kibana, open the menu and go to Stack Management > Index Lifecycle Policies.
  2. Create a new 60 day retention lifecycle policy.
    1. Click Create policy.
    2. Name the new policy mi-log-general-policy.
    3. In the Hot phase section, switch from Keep data in this phase forever to Delete data after this phase.
    4. In the Delete phase panel below, set Move data into phase when to 60 days. This ensures that index data is retained for 60 days, after which it will be removed.
    5. Click Save policy at the bottom of the page.
  3. Create a second policy called mi-log-audit-policy with no changes to the default settings.

To use a policy that triggers the rollover action, you need to configure the policy in the index template used to create each new index. You specify the name of the policy and the alias used to reference the rolling indices.

  1. Go to Index Management > Index Templates.
  2. Search for 'mi' to find the default mi-log template.
  3. Clone the mi-log template: click Actions > Clone.
    1. On the Logistics tab, set the name to mi-log-audit-template.
    2. In the Index patterns field, add mi-log-audit.
    3. Clear the Priority field.
    4. Go to the Index Settings tab and set the lifecycle name to mi-log-audit-policy: 
      {
  "index": {
    "lifecycle": {
      "name": "mi-log-audit-policy"
    },
...
    5. Go to the Review template tab, review the settings, then click Create template.
  4. Clone the mi-log template again: click Actions > Clone.
    1. On the Logistics tab, set the name to mi-log-general-template.
    2. In the Index patterns field, add mi-log and mi-log-activity.


    3. Clear the Priority field.
    4. Go to the Index Settings tab and set the lifecycle name to mi-log-general-policy: 
      {
  "index": {
    "lifecycle": {
      "name": "mi-log-general-policy"
    },
...
    5. Go to the Review template tab, review the settings, then click Create template.
  5. Delete the original mi-log template.