Dec-21-2017, 09:57 AM
I need help with following assignment, I am trying to do it for a week now but couldn't as I am new in phyton. Can any body help to create phyton script with these functionality:
Import a.Json
Extract data
Conditional action on data
Combine data and flatten it
Export as.csv
====project details====
Grab all _id
Based on _type extract the following data from those indexes
"_type": "visualization"
"title": " " --> CSV_column=Visualisation-title
kibanaSavedObjectMeta": {"searchSourceJSON": {"index": ""} } --> CSV_column=Visualisation-index
"savedSearchId": " " --> variable for mapping
"_id": " " --> CSV_column=Search-ID
"_type": "search"
"title": "" --> CSV_column=Search-title
kibanaSavedObjectMeta": {"searchSourceJSON": {"index": ""} } --> CSV_column=Index-name
"_id": " " --> CSV_column=Dashboard-id
"_type": "dashboard",
"title": "" --> CSV_column=Dashboard-Title
"panelsJSON": "id\": <- iterate for all values --> variable for mapping
savedSearchId in the _type visualisation can be used to get the name for Index-name if this is not available in the searchSourceJSON.
It is possible to have the _type visualisation without an accompanying Index-name, either directly or via the savedSearchId. So take that into account.
Create CSV, map data visualization_id & object from dashboard panelsJSON id and/or from visualization savedSearchId matching _id from _type "search"
the 4-6th line in the output.CSV are examples of data not available in the current input.json.
====project details====
====output.csv====
Visualisation-ID, Visualisation-title, Index-name, Dashboard-id, Dashboard-Title, Search-ID, Search-title
"51a99350-dad0-","title of this visualisation-03","index-name-02","b40bc620-dad1-","title of this dashboard",,,
"5300a260-dad1-","title of this visualisation-04","index-name-01","b40bc620-dad1-","title of this dashboard","5fea1790-dacc-","title of this saved search-01"
"53sdf950-dda5-","title of this visualisation-08",,b40bc620-dad1-","title of this dashboard",,
,,,b40bc620-dad1-","title of this dashboard",,
,,,,,"5f235490-dacc-","title of this saved search-02"
====output.csv====
====input.json====
[
{
"_id": "b40bc620-dad1-",
"_type": "dashboard",
"_source": {
"title": "title of this dashboard",
"hits": 0,
"description": "description of dashboard",
"panelsJSON": "[{\"col\":1,\"id\":\"22e86ec0-dac5-\",\"panelIndex\":1,\"row\":11,\"size_x\":12,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"af5d8150-dac6-\",\"panelIndex\":2,\"row\":7,\"size_x\":12,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"51a99350-dad0-\",\"panelIndex\":3,\"row\":1,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"5300a260-dad1-\",\"panelIndex\":4,\"row\":4,\"size_x\":12,\"size_y\":3,\"type\":\"visualization\"},{\"size_x\":6,\"size_y\":3,\"panelIndex\":5,\"type\":\"visualization\",\"id\":\"544290e0-dad4-\",\"col\":7,\"row\":1}]",
"optionsJSON": {\"...not relevant"}
"uiStateJSON": "{\"...not relevant"}",
"version": 1,
"timeRestore": false,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"...not relevant"}"
}
}
},
{
"_id": "22e86ec0-dac5-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-01",
"visState": "{\"...not relevant"}",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"index-name-01\",\"query\":{\"match_all\":{}},\"filter\":[]}"
}
}
},
{
"_id": "af5d8150-dac6-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-02",
"visState": "{\"...not relevant"}",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"index-name-01\",\"query\":{\"match_all\":{}},\"filter\":[]}"
}
}
},
{
"_id": "51a99350-dad0-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-03",
"visState": "{\"...not relevant"}",
"uiStateJSON": "{\"vis\":{\"defaultColors\":{\"0 - 100\":\"rgb(0,104,55)\"}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[],\"index\":\"index-name-02\",\"highlightAll\":true,\"version\":true,\"query\":{\"match_all\":{}}}"
}
}
},
{
"_id": "5300a260-dad1-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-04",
"visState": {\"...not relevant"},
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"description": "",
"savedSearchId": "5fea1790-dacc-",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
},
{
"_id": "544290e0-dad4-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-05",
"visState": "{\"...not relevant"},
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"description": "",
"savedSearchId": "5fea1790-dacc-",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
},
{
"_id": "5fea1790-dacc-",
"_type": "search",
"_source": {
"title": "title of this saved search-01",
"description": "",
"hits": 0,
"columns": [
"_source"
],
"sort": [
"_score",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"index-name-01\", "...not relevant"}
}
}
}
]
Import a.Json
Extract data
Conditional action on data
Combine data and flatten it
Export as.csv
====project details====
Grab all _id
Based on _type extract the following data from those indexes
"_type": "visualization"
"title": " " --> CSV_column=Visualisation-title
kibanaSavedObjectMeta": {"searchSourceJSON": {"index": ""} } --> CSV_column=Visualisation-index
"savedSearchId": " " --> variable for mapping
"_id": " " --> CSV_column=Search-ID
"_type": "search"
"title": "" --> CSV_column=Search-title
kibanaSavedObjectMeta": {"searchSourceJSON": {"index": ""} } --> CSV_column=Index-name
"_id": " " --> CSV_column=Dashboard-id
"_type": "dashboard",
"title": "" --> CSV_column=Dashboard-Title
"panelsJSON": "id\": <- iterate for all values --> variable for mapping
savedSearchId in the _type visualisation can be used to get the name for Index-name if this is not available in the searchSourceJSON.
It is possible to have the _type visualisation without an accompanying Index-name, either directly or via the savedSearchId. So take that into account.
Create CSV, map data visualization_id & object from dashboard panelsJSON id and/or from visualization savedSearchId matching _id from _type "search"
the 4-6th line in the output.CSV are examples of data not available in the current input.json.
====project details====
====output.csv====
Visualisation-ID, Visualisation-title, Index-name, Dashboard-id, Dashboard-Title, Search-ID, Search-title
"51a99350-dad0-","title of this visualisation-03","index-name-02","b40bc620-dad1-","title of this dashboard",,,
"5300a260-dad1-","title of this visualisation-04","index-name-01","b40bc620-dad1-","title of this dashboard","5fea1790-dacc-","title of this saved search-01"
"53sdf950-dda5-","title of this visualisation-08",,b40bc620-dad1-","title of this dashboard",,
,,,b40bc620-dad1-","title of this dashboard",,
,,,,,"5f235490-dacc-","title of this saved search-02"
====output.csv====
====input.json====
[
{
"_id": "b40bc620-dad1-",
"_type": "dashboard",
"_source": {
"title": "title of this dashboard",
"hits": 0,
"description": "description of dashboard",
"panelsJSON": "[{\"col\":1,\"id\":\"22e86ec0-dac5-\",\"panelIndex\":1,\"row\":11,\"size_x\":12,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"af5d8150-dac6-\",\"panelIndex\":2,\"row\":7,\"size_x\":12,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"51a99350-dad0-\",\"panelIndex\":3,\"row\":1,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"5300a260-dad1-\",\"panelIndex\":4,\"row\":4,\"size_x\":12,\"size_y\":3,\"type\":\"visualization\"},{\"size_x\":6,\"size_y\":3,\"panelIndex\":5,\"type\":\"visualization\",\"id\":\"544290e0-dad4-\",\"col\":7,\"row\":1}]",
"optionsJSON": {\"...not relevant"}
"uiStateJSON": "{\"...not relevant"}",
"version": 1,
"timeRestore": false,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"...not relevant"}"
}
}
},
{
"_id": "22e86ec0-dac5-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-01",
"visState": "{\"...not relevant"}",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"index-name-01\",\"query\":{\"match_all\":{}},\"filter\":[]}"
}
}
},
{
"_id": "af5d8150-dac6-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-02",
"visState": "{\"...not relevant"}",
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"}}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"index-name-01\",\"query\":{\"match_all\":{}},\"filter\":[]}"
}
}
},
{
"_id": "51a99350-dad0-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-03",
"visState": "{\"...not relevant"}",
"uiStateJSON": "{\"vis\":{\"defaultColors\":{\"0 - 100\":\"rgb(0,104,55)\"}}}",
"description": "",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[],\"index\":\"index-name-02\",\"highlightAll\":true,\"version\":true,\"query\":{\"match_all\":{}}}"
}
}
},
{
"_id": "5300a260-dad1-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-04",
"visState": {\"...not relevant"},
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"description": "",
"savedSearchId": "5fea1790-dacc-",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
},
{
"_id": "544290e0-dad4-",
"_type": "visualization",
"_source": {
"title": "title of this visualisation-05",
"visState": "{\"...not relevant"},
"uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
"description": "",
"savedSearchId": "5fea1790-dacc-",
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\":[]}"
}
}
},
{
"_id": "5fea1790-dacc-",
"_type": "search",
"_source": {
"title": "title of this saved search-01",
"description": "",
"hits": 0,
"columns": [
"_source"
],
"sort": [
"_score",
"desc"
],
"version": 1,
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"index\":\"index-name-01\", "...not relevant"}
}
}
}
]