class: center, middle, inverse, title-slide .title[ # Wikidata for data journalism (with R and ‘tidywikidatar’) ] .subtitle[ ## Dataharvest 2022 ] .author[ ### Giorgio Comai ] .institute[ ### (OBCT/CCI - EDJNet) ] .date[ ### 22 May 2022 ] --- # Two big questions ### 1. how to you get Wikidata to throw at you the data you need ### 2. how to retrieve Wikidata's data in a way that won't give you headaches --- # Access Wikidata Web front-end, nice to read: https://www.wikidata.org/wiki/Q162022 SPARQL: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/queries/examples API version https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q162022 - machine readable - nested lists, structure changes depending on values --- # What's the matter (for R users)? - R users mostly hate nested lists - some probably also hate SPARQL, but even more simply don't know much about SPARQL - if you don't know what to expect, it's a pain to process data - existing tools are not fit for the iterative data analysis process that is at the core of data journalism - re-running an analysis with minor changes is a very common part of the workflow... without built-in caching, this can be painfully slow --- # `tidywikidatar` <div style ="float: right;" ><img src="img/tidywikidatar_logo.png" style = "width:256px;"></img></div> Check out website with documentation and examples: https://edjnet.github.io/tidywikidatar/ [](https://cran.r-project.org/package=tidywikidatar) [](https://r-pkg.org/pkg/tidywikidatar) [](https://r-pkg.org/pkg/tidywikidatar) - everything in tabular format - one row, one piece of information - easy local caching - easy integration with `dplyr` piped routines - get image credits from WikiMedia commons - include Wikipedia in the exploration, or use it as a starting point --- class: middle, center background-size: contain # `tidywikidatar` # The basics --- # Enable local caching ```r library(dplyr, warn.conflicts = FALSE) library("tidywikidatar") tw_enable_cache() tw_set_cache_folder( path = fs::path(fs::path_home_r(), "R", "tw_data")) tw_set_language(language = "en") tw_create_cache_folder(ask = FALSE) ``` --- # Or e.g. MySQL ```r library(dplyr, warn.conflicts = FALSE) library("tidywikidatar") tw_enable_cache(SQLite = FALSE) tw_set_cache_db(driver = "MySQL", host = "localhost", port = 3306, database = "tidywikidatar", user = "secret_username", pwd = "secret_password") ``` --- # Get an item ```r tw_search("Mechelen") ``` <div id="htmlwidget-49c48c11dd11e4929dae" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-49c48c11dd11e4929dae">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10"],["Q162022","Q100020","Q21765751","Q406287","Q3597347","Q16025171","Q65044254","Q103956883","Q542231","Q585198"],["Mechelen","Mechelen","Mechelen","Mechelen","13293 Mechelen","Mechelen","Mechelen","Mechelen","Mechelen concentration camp","R6 road"],["city in the province of Antwerp, Belgium","town in Gulpen-Wittem, Netherlands","town in Mechelen municipality, Belgium","Wikimedia disambiguation page","asteroid","minesweeper","Belgian political subdivision","encyclopedia article","former Nazi transit camp and a current museum in Belgium","ring road of Mechelen, Belgium"]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>id<\/th>\n <th>label<\/th>\n <th>description<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":5,"lengthMenu":[3,5,10,15,20],"columnDefs":[{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script> --- # Get an item ```r tw_search("Mechelen") %>% slice(1) %>% tw_get() ``` <div id="htmlwidget-73c6cc24a29bde5e3690" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-73c6cc24a29bde5e3690">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110"],["Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022","Q162022"],["alias_en","alias_en","description_en","label_en","P1036","P10397","P1082","P1082","P1082","P1296","P131","P131","P131","P1343","P1343","P1448","P1449","P1456","P1456","P1456","P1464","P1465","P1549","P1549","P1566","P1567","P1667","P17","P1705","P1792","P18","P18","P190","P190","P190","P190","P1997","P2046","P2046","P206","P206","P206","P206","P206","P213","P214","P2184","P227","P237","P242","P244","P268","P281","P281","P281","P281","P2924","P31","P31","P3219","P3365","P36","P3608","P373","P3984","P402","P41","P417","P4342","P443","P4672","P47","P47","P47","P47","P47","P47","P473","P473","P527","P527","P527","P527","P5573","P5785","P5982","P6","P625","P646","P6766","P691","P7818","P7850","P7859","P7867","P793","P8168","P8189","P8313","P8406","P8519","P856","P8989","P910","P94","P948","P9757","P982","P9957","sitelink_enwiki"],["Mechlin","Malines","city in the province of Antwerp, Belgium","Mechelen","2--49322","TO0L002125","+86304","+86616","+86921","0039489","Q90905","Q638575","Q1206596","Q3181656","Q20078554","Mechelen","Dijlestad","Q2528205","Q3252127","Q2738415","Q8063472","Q9219001","Mechelaar","Mechlinian","2791538","12025","7008736","Q31","Mechelen","Q7920559","Mechelen Sint-Rombouts.JPG","Stadhuis Mechelen, 2013.jpg","Q590849","Q30002","Q83324","Q9844","110294412324669","+33.71","+65.80","Q845920","Q934226","Q1545935","Q1630799","Q2533815","0000 0004 0456 8699","312798307","Q2354996","4074654-9","Q14517589","Mechelen Antwerp Belgium Map.svg","n80086435","119588361","2800","2801","2811","2812","2210306","Q15273785","Q493522","malines","malines","Q21765751","BE0207499430","Mechelen","mechelen","412864","Flag of Mechlin.svg","Q965463","Mechelen","Nl-Mechelen.ogg","29053653-b038-4e4b-9d46-57e46bee6094","Q696578","Q943446","Q695278","Q179249","Q746078","Q911995","015","03","Q2856042","Q2498309","Q2795382","Q2434402","3636","951863620","+383","Q15993","51.0280555555556,4.48027777777778","/m/014h9r","101748091","ge523280","Malines","T84-677","lccn-n80086435","Q84026608","Q2726440","Q88573","987007552496805171","Mechelen","T056326","764","https://www.mechelen.be","Q104597522","Q7481161","Wapen Mechelen.svg","Mechelen Wikivoyage Banner.png","568","f05e07a9-0ea2-4b8e-9538-370456752089","542","Mechelen"],[null,null,null,null,"deprecated","normal","normal","normal","preferred","normal","preferred","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","preferred","normal","normal","normal","normal","normal","deprecated","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal",null]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>id<\/th>\n <th>property<\/th>\n <th>value<\/th>\n <th>rank<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":5,"lengthMenu":[3,5,10,15,20],"columnDefs":[{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script> --- ```r tw_search("Mechelen") %>% slice(1) %>% tw_get() %>% * tw_label() ``` <div id="htmlwidget-432915894560232ffad7" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-432915894560232ffad7">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110"],["Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen","Mechelen"],[null,null,null,null,"Dewey Decimal Classification","SBN place ID","population","population","population","Gran Enciclopèdia Catalana ID","located in the administrative territorial entity","located in the administrative territorial entity","located in the administrative territorial entity","described by source","described by source","official name","nickname","list of monuments","list of monuments","list of monuments","category for people born here","category for people who died here","demonym","demonym","GeoNames ID","NIS/INS code","Getty Thesaurus of Geographic Names ID","country","native label","category of associated people","image","image","twinned administrative body","twinned administrative body","twinned administrative body","twinned administrative body","Facebook Places ID","area","area","located in or next to body of water","located in or next to body of water","located in or next to body of water","located in or next to body of water","located in or next to body of water","ISNI","VIAF ID","history of topic","GND ID","coat of arms","locator map image","Library of Congress authority ID","Bibliothèque nationale de France ID","postal code","postal code","postal code","postal code","Great Russian Encyclopedia Online ID","instance of","instance of","Encyclopædia Universalis ID","Treccani ID","capital","EU VAT number","Commons category","subreddit","OpenStreetMap relation ID","flag image","patron saint","Store norske leksikon ID","pronunciation audio","EMLO location ID","shares border with","shares border with","shares border with","shares border with","shares border with","shares border with","local dialing code","local dialing code","has part or parts","has part or parts","has part or parts","has part or parts","archINFORM location ID","EU Research participant ID","annual number of weddings","head of government","coordinate location","Freebase ID","Who's on First ID","NKCR AUT ID","French Vikidia ID","Joconde location ID","WorldCat Identities ID","category for maps","significant event","FactGrid item ID","National Library of Israel J9U ID","Den Store Danske ID","Grove Art Online ID","RKD thesaurus ID","official website","category for the view of the item","topic's main category","coat of arms image","page banner","Schoenberg Database of Manuscripts place ID","MusicBrainz area ID","museum-digital place ID",null],["Mechlin","Malines","city in the province of Antwerp, Belgium","Mechelen","2--49322","TO0L002125","+86304","+86616","+86921","0039489","Arrondissement of Mechelen","Lordship of Mechelen","Deux-Nèthes","The Nuttall Encyclopædia","Great Soviet Encyclopedia (1926–1947)","Mechelen","Dijlestad",null,null,null,"Category:Births in Mechelen","Category:Deaths in Mechelen","Mechelaar","Mechlinian","2791538","12025","7008736","Belgium","Mechelen","Category:People from Mechelen","Mechelen Sint-Rombouts.JPG","Stadhuis Mechelen, 2013.jpg","Arvada","Chengdu","Sibiu","Helmond","110294412324669","+33.71","+65.80","Senne","Dyle / Dijle","Nete","Channel Leuven-Mechelen (Dijle)",null,"0000 0004 0456 8699","312798307",null,"4074654-9","coat of arms of Mechelen","Mechelen Antwerp Belgium Map.svg","n80086435","119588361","2800","2801","2811","2812","2210306","Belgian municipality with the title of city","municipality of Belgium","malines","malines","Mechelen","BE0207499430","Mechelen","mechelen","412864","Flag of Mechlin.svg","Rumbold of Mechelen","Mechelen","Nl-Mechelen.ogg","29053653-b038-4e4b-9d46-57e46bee6094","Sint-Katelijne-Waver","Rumst","Willebroek","Zemst","Boortmeerbeek","Kapelle-op-den-Bos","015","03","Walem","Muizen","Leest","Heffen","3636","951863620","+383","Bart Somers","51.0280555555556,4.48027777777778","/m/014h9r","101748091","ge523280","Malines","T84-677","lccn-n80086435","Category:Maps of Mechelen","Spanish Fury at Mechelen","Joseph Melcher","987007552496805171","Mechelen","T056326","764","https://www.mechelen.be","Category:Views of Mechelen","Category:Mechelen","Wapen Mechelen.svg","Mechelen Wikivoyage Banner.png","568","f05e07a9-0ea2-4b8e-9538-370456752089","542","Mechelen"],[null,null,null,null,"deprecated","normal","normal","normal","preferred","normal","preferred","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","preferred","normal","normal","normal","normal","normal","deprecated","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal","normal",null]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>id<\/th>\n <th>property<\/th>\n <th>value<\/th>\n <th>rank<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":5,"lengthMenu":[3,5,10,15,20],"columnDefs":[{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script> --- # Get a specific property ```r tw_search("European Union") %>% slice(1) %>% tw_get_property(p = "P31") # instance of ``` <div id="htmlwidget-ebc0ef81e2b8eaa99590" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-ebc0ef81e2b8eaa99590">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6"],["Q458","Q458","Q458","Q458","Q458","Q458"],["P31","P31","P31","P31","P31","P31"],["Q4120211","Q3623811","Q1335818","Q7210356","Q1048835","Q170156"],["normal","normal","normal","normal","normal","normal"]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>id<\/th>\n <th>property<\/th>\n <th>value<\/th>\n <th>rank<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":5,"lengthMenu":[3,5,10,15,20],"columnDefs":[{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script> --- # Get a specific property ```r tw_search("European Union") %>% slice(1) %>% tw_get_property(p = "P150") %>% # contains administrative territorial entity * tw_label() ``` <div id="htmlwidget-e1d7b71a6319aeaafe9a" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-e1d7b71a6319aeaafe9a">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28"],["European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union","European Union"],["contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity","contains administrative territorial entity"],["Belgium","Germany","France","Italy","Luxembourg","Kingdom of the Netherlands","Denmark","Republic of Ireland","Greece","Portugal","Spain","Finland","Austria","Sweden","Estonia","Latvia","Lithuania","Malta","Poland","Slovakia","Slovenia","Czech Republic","Hungary","Cyprus","Bulgaria","Romania","Croatia","United Kingdom"],["preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","preferred","normal"]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>id<\/th>\n <th>property<\/th>\n <th>value<\/th>\n <th>rank<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":5,"lengthMenu":[3,5,10,15,20],"columnDefs":[{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script> --- # What about qualifiers? e.g. when did member states join the EU? https://www.wikidata.org/wiki/Q458#P150 ```r tw_get_qualifiers(id = "Q458", # European Union p = "P150") # contains administrative territorial entity ``` ``` ## # A tibble: 37 × 8 ## id property qualifier_id qualifier_prope… qualifier_value qualifier_value… ## <chr> <chr> <chr> <chr> <chr> <chr> ## 1 Q458 P150 Q31 P580 +1958-01-01T00… time ## 2 Q458 P150 Q183 P580 +1958-01-01T00… time ## 3 Q458 P150 Q142 P580 +1958-01-01T00… time ## 4 Q458 P150 Q142 P1012 Q3769 wikibase-entity… ## 5 Q458 P150 Q142 P1012 Q17012 wikibase-entity… ## 6 Q458 P150 Q142 P1012 Q17054 wikibase-entity… ## 7 Q458 P150 Q142 P1012 Q17063 wikibase-entity… ## 8 Q458 P150 Q142 P1012 Q17070 wikibase-entity… ## 9 Q458 P150 Q142 P1012 Q126125 wikibase-entity… ## 10 Q458 P150 Q38 P580 +1958-01-01T00… time ## # … with 27 more rows, and 2 more variables: rank <chr>, set <dbl> ``` --- # What about qualifiers? ```r tw_get_qualifiers(id = "Q458", # European Union p = "P150") %>% # contains administrative territorial entity filter(qualifier_property == "P580") %>% # keep only "start time" * transmute(country = tw_get_label(qualifier_id), * start_time = qualifier_value) ``` ``` ## # A tibble: 28 × 2 ## country start_time ## <chr> <chr> ## 1 Belgium +1958-01-01T00:00:00Z ## 2 Germany +1958-01-01T00:00:00Z ## 3 France +1958-01-01T00:00:00Z ## 4 Italy +1958-01-01T00:00:00Z ## 5 Luxembourg +1958-01-01T00:00:00Z ## 6 Kingdom of the Netherlands +1958-01-01T00:00:00Z ## 7 Denmark +1973-01-01T00:00:00Z ## 8 Republic of Ireland +1973-01-01T00:00:00Z ## 9 Greece +1981-01-01T00:00:00Z ## 10 Portugal +1986-01-01T00:00:00Z ## # … with 18 more rows ``` - for more, check `tw_get_property_with_details()` --- ### Dealing with multiple results when only one is needed Easy questions can be difficult: in which country is London? ```r tibble::tibble(city_qid = c("Q84")) %>% dplyr::mutate(city_label = tw_get_label(city_qid), country_qid = tw_get_p(id = city_qid, p = "P17")) %>% tidyr::unnest(country_qid) %>% mutate(country = tw_get_label(country_qid)) ``` ``` ## # A tibble: 8 × 4 ## city_qid city_label country_qid country ## <chr> <chr> <chr> <chr> ## 1 Q84 London Q2277 Roman Empire ## 2 Q84 London Q110888 Kingdom of Essex ## 3 Q84 London Q105092 Kingdom of Mercia ## 4 Q84 London Q105313 Kingdom of Wessex ## 5 Q84 London Q179876 Kingdom of England ## 6 Q84 London Q161885 Great Britain ## 7 Q84 London Q174193 United Kingdom of Great Britain and Ireland ## 8 Q84 London Q145 United Kingdom ``` --- ### Dealing with multiple results when only one is needed - keeping first result is tricky - keeping only preferred may still give more than one result - people who love tabular data often want just one result, that needs to be "good enough" ```r tibble::tibble(city_qid = c("Q84", "Q220")) %>% dplyr::mutate(city_label = tw_get_label(city_qid), country_qid = tw_get_p(id = city_qid, p = "P17", * preferred = TRUE, * # latest_start_time = TRUE, * only_first = TRUE)) %>% dplyr::mutate(country_label = tw_get_label(country_qid)) ``` ``` ## # A tibble: 2 × 4 ## city_qid city_label country_qid country_label ## <chr> <chr> <chr> <chr> ## 1 Q84 London Q145 United Kingdom ## 2 Q220 Rome Q38 Italy ``` --- # Less typing ```r tw_qid_meps %>% head() %>% mutate(name = tw_get_label(id), * pob = tw_get_p1(id = id, p = "P19")) %>% mutate(pob_label = tw_get_label(id = pob), * pob_coordinates = tw_get_p1(id = pob, p = "P625")) ``` ``` ## # A tibble: 6 × 5 ## id name pob pob_label pob_coordinates ## <chr> <chr> <chr> <chr> <chr> ## 1 Q157 François Hollande Q30974 Rouen 49.44305555555… ## 2 Q329 Nicolas Sarkozy Q90 Paris 48.85694444444… ## 3 Q1220 Giorgio Napolitano Q2634 Naples 40.83333333333… ## 4 Q1275 Gladwyn Jebb Q163 Yorkshire 53.95833333333… ## 5 Q2105 Jacques Chirac Q238723 5th arrondissement of Paris 48.84722222222… ## 6 Q2512 Kurt Georg Kiesinger Q7019 Albstadt 48.21194444444… ``` # More properties, all at once ```r tw_qid_meps %>% head() %>% * tw_get_p_wide(p = c("P21", "P27", "P569"), * property_label_as_column_name = FALSE, * label = FALSE, * only_first = TRUE) ``` ``` ## # A tibble: 6 × 4 ## id P21 P27 P569 ## <chr> <chr> <chr> <chr> ## 1 Q157 Q6581097 Q142 +1954-08-12T00:00:00Z ## 2 Q329 Q6581097 Q142 +1955-01-28T00:00:00Z ## 3 Q1220 Q6581097 Q38 +1925-06-29T00:00:00Z ## 4 Q1275 Q6581097 Q145 +1900-04-25T00:00:00Z ## 5 Q2105 Q6581097 Q142 +1932-11-29T00:00:00Z ## 6 Q2512 Q6581097 Q183 +1904-04-06T00:00:00Z ``` ```r tw_qid_meps %>% head() %>% * tw_get_p_wide(p = c("P21", "P27", "P569"), * property_label_as_column_name = TRUE, * label = TRUE, * only_first = TRUE) ``` ``` ## # A tibble: 6 × 5 ## id label sex_or_gender country_of_citizenship date_of_birth ## <chr> <chr> <chr> <chr> <chr> ## 1 Q157 François Hollande male France +1954-08-12T0… ## 2 Q329 Nicolas Sarkozy male France +1955-01-28T0… ## 3 Q1220 Giorgio Napolitano male Italy +1925-06-29T0… ## 4 Q1275 Gladwyn Jebb male United Kingdom +1900-04-25T0… ## 5 Q2105 Jacques Chirac male France +1932-11-29T0… ## 6 Q2512 Kurt Georg Kiesinger male Germany +1904-04-06T0… ``` --- class: middle, center background-size: contain # Different entry points --- ### Search - `tw_search()` - search strings ### Query - `tw_query()` - simple queries based on property/value couples - `tw_get_all_with_p()` - get all items that have a given property, irrespective of their value ### Based on Wikipedia - `tw_get_wikipedia_page_links()` - Get all Wikidata Q identifiers of all Wikipedia pages linked to input - `tw_get_wikipedia_page_section_links()` - All identifiers found in a specific section of a Wikipedia page --- class: middle, center background-size: contain # An example starting from Wikipedia --- # Election of the President of the Republic in Italy <div style ="float: right;" ><img src="img/IV_scrutinio.png" style = "width:256px;"></img></div> - [Election of the President of the Republic in Italy](https://it.wikipedia.org/wiki/Elezione_del_Presidente_della_Repubblica_Italiana_del_2022) - the electoral college can vote literally for whoever they like - the list ends up including very different candidates, from respected intellectuals to football players and porn actors - almost all of them with one thing in common: they are on Wikidata, but Wikidata does not know they have something in common. --- ## Wikidata identifiers .pull-left[ Take a single section: ```r df <- tw_get_wikipedia_page_section_links( title = "Elezione del Presidente della Repubblica Italiana del 2022", section_title = "IV scrutinio", language = "it") df %>% select(wikipedia_title, qid) ``` ] .pull-right[ |wikipedia_title |qid | |:--------------------|:---------| |Adnkronos |Q359875 | |Alberto Airola |Q14636378 | |Aldo Giannuli |Q3609233 | |Alessandro Altobelli |Q346945 | |Alessandro Barbero |Q960451 | |Carlo Nordio |Q19357364 | |Dino Zoff |Q180661 | |Domenico De Masi |Q3713005 | ] --- # Find out more ```r pob_df <- df %>% select(qid) %>% mutate(name = tw_get_label(qid)) %>% mutate(place_of_birth_id = tw_get_p(id = qid, p = "P19",only_first = TRUE)) %>% mutate(place_of_birth = tw_get_label(place_of_birth_id)) %>% mutate(place_of_birth_coordinates = tw_get_p(id = place_of_birth_id, p = "P625", only_first = TRUE)) pob_df ``` ``` ## # A tibble: 36 × 5 ## qid name place_of_birth_… place_of_birth place_of_birth_… ## <chr> <chr> <chr> <chr> <chr> ## 1 Q359875 Adnkronos <NA> <NA> <NA> ## 2 Q14636378 Alberto Airola Q9474 Moncalieri 45,7.683333 ## 3 Q3609233 Aldo Giannuli Q3519 Bari 41.125277777778… ## 4 Q346945 Alessandro Altobe… Q128211 Sonnino 41.414458333333… ## 5 Q960451 Alessandro Barbero Q495 Turin 45.066666666667… ## 6 Q19357364 Carlo Nordio Q5475 Treviso 45.672219444444… ## 7 Q180661 Dino Zoff Q53131 Mariano del F… 45.916666666667… ## 8 Q3713005 Domenico de Masi Q277969 Rotello 41.7475,15.0041… ## 9 Q3723207 Elisabetta Belloni Q220 Rome 41.893055555556… ## 10 Q726247 Franco Grillini Q94979 Pianoro 44.383333333333… ## # … with 26 more rows ``` --- ### Here they are on a map .pull-left[ ```r pob_sf <- pob_df %>% tidyr::separate( col = place_of_birth_coordinates, into = c("pob_latitude","pob_longitude"), sep = ",", remove = TRUE, convert = TRUE) %>% filter(is.na(pob_latitude)==FALSE) %>% sf::st_as_sf(coords = c("pob_longitude", "pob_latitude"), crs = 4326) library("ggplot2") pop_gg <- ggplot() + geom_sf(data = ll_get_nuts_it(level = 3, no_check_certificate = TRUE)) + geom_sf(data = pob_sf, colour = "deeppink4") + theme_minimal() ``` ``` ## ℹ Source: https://www.istat.it/it/archivio/222527 ## ℹ Istat (CC-BY) ``` ] .pull-right[ <!-- --> ] --- ### All the usual things we expect from Wikidata .pull-left[ ```r occupation_df <- df %>% pull(qid) %>% tw_get_property(p = "P31") %>% # get "instance of" filter(value == "Q5") %>% # keep only humans tw_get_property(p = "P106") %>% # get occupation # filter(value!="Q82955") %>% # exclude politicians group_by(value) %>% count(sort = TRUE) %>% ungroup() %>% transmute(occupation = tw_get_label(value), n) ``` ] .pull-right[ |occupation | n| |:----------------------------|--:| |politician | 71| |university teacher | 19| |judge | 16| |lawyer | 14| |jurist | 10| |deputy chairperson | 9| |basketball player | 9| |physician | 9| |clerk | 9| |journalist | 7| |sociologist | 5| |association football player | 5| |economist | 4| |psychologist | 4| |academic | 4| |banker | 4| |association football manager | 3| |historian | 2| |writer | 2| |high civil servant | 1| |essayist | 1| |music director | 1| |professor | 1| |entrepreneur | 1| |media proprietor | 1| |conductor | 1| |diplomat | 1| |film director | 1| |sports executive | 1| |radio personality | 1| |medievalist | 1| |theatrical director | 1| |film critic | 1| |magistrate | 1| |musician | 1| |television presenter | 1| ] --- ### And other things useful for data visualisation and interactive interfaces, e.g. quick access to images... ```r president_df <- tw_search("Sergio Mattarella") %>% tw_filter_first(p = "P31", q = "Q5") president_df %>% tw_get_image() ``` ``` ## # A tibble: 3 × 2 ## id image ## <chr> <chr> ## 1 Q3956186 Sergio Mattarella Presidente della Repubblica Italiana.jpg ## 2 Q3956186 Sergio Mattarella Presidente della Repubblica Italiana.jpg ## 3 Q3956186 Sergio Mattarella Presidente della Repubblica Italiana.jpg ``` ```r president_df %>% tw_get_image(format = "embed", width = 300) %>% pull(image) ``` ``` ## [1] "https://commons.wikimedia.org/w/index.php?title=Special:Redirect/file/Sergio Mattarella Presidente della Repubblica Italiana.jpg&width=300" ## [2] "https://commons.wikimedia.org/w/index.php?title=Special:Redirect/file/Sergio Mattarella Presidente della Repubblica Italiana.jpg&width=300" ## [3] "https://commons.wikimedia.org/w/index.php?title=Special:Redirect/file/Sergio Mattarella Presidente della Repubblica Italiana.jpg&width=300" ``` --- ### ...with metadata and credits line ```r president_df %>% tw_get_image_metadata() %>% tidyr::pivot_longer(cols = -1, names_to = "property", values_transform = as.character) ``` ``` ## # A tibble: 18 × 3 ## id property value ## <chr> <chr> <chr> ## 1 Q3956186 image_filename "Sergio Mattarella Presidente della Repu… ## 2 Q3956186 object_name "Sergio Mattarella Presidente della Repu… ## 3 Q3956186 image_description "Official picture of the <a href=\"https… ## 4 Q3956186 categories "Attribution only license|Images from th… ## 5 Q3956186 assessments "" ## 6 Q3956186 credit "<a rel=\"nofollow\" class=\"external te… ## 7 Q3956186 artist "Unknown author<span style=\"display: no… ## 8 Q3956186 permission <NA> ## 9 Q3956186 license_short_name "Attribution" ## 10 Q3956186 license_url <NA> ## 11 Q3956186 license <NA> ## 12 Q3956186 usage_terms <NA> ## 13 Q3956186 attribution_required <NA> ## 14 Q3956186 copyrighted <NA> ## 15 Q3956186 restrictions "personality" ## 16 Q3956186 date_time "2022-05-14 10:47:24" ## 17 Q3956186 date_time_original "2022-01-29" ## 18 Q3956186 commons_metadata_extension "1.2" ``` --- ### Back and forth between Wikidata and Wikipedia This gets the Q identifier of all pages linked from a the Wikipedia page of a given Q identifier. Easy peasy :-) ```r president_df %>% tw_get_wikipedia(language = "it") %>% # gets url of Wikipedia page from QID tw_get_wikipedia_page_links(language = "it") %>% select(wikipedia_title, qid) ``` ``` ## # A tibble: 491 × 2 ## wikipedia_title qid ## <chr> <chr> ## 1 Fabio Vander <NA> ## 2 Ordine per Meriti Eccezionali <NA> ## 3 Discussioni template:Capi di Stato d'Europa <NA> ## 4 1941 Q5231 ## 5 1987 Q2429 ## 6 1989 Q2425 ## 7 1990 Q2064 ## 8 1998 Q2089 ## 9 1999 Q2091 ## 10 2001 Q1988 ## # … with 481 more rows ``` --- # Starting from strings <img src="img/example_street_names.png" width="1280px" /> --- class: middle, center background-size: contain # Integration with interactive interfaces --- class: middle, center background-image: url(img/screenshot_mapping_diversity_01.png) background-size: contain --- class: middle, center background-image: url(img/screenshot_mapping_diversity_02.png) background-size: contain --- class: middle, center background-size: contain # A couple of examples of practical use cases --- ### Olympics 2020 medalists by place of birth https://github.com/EDJNet/olympics2020nuts <iframe src="https://edjnet.github.io/olympics2020nuts/medalists_map.html" width="100%" height="400px" data-external="1"></iframe> --- ### Main air routes that could be travelled by train Wikidata used for: defining city hubs for airports, getting coordinates of airports (for excluding those on islands), use unique identifiers for merging with train dataset [https://edjnet.github.io/european_routes/](https://edjnet.github.io/european_routes/) <img src="img/air_train_routes_germany.png" width="640px" /> --- ### Mapping diversity https://medium.com/european-data-journalism-network/finding-gendered-street-names-a-step-by-step-walkthrough-with-r-7608c2d36a77 <iframe src="https://mappingdiversity.eu/italy/bologna/" width="100%" height="400px" data-external="1"></iframe> # Issues and next steps --- # General issues - if you are processing many thousands of items, the current approach can be very slow when run for the first time (near instant thanks to caching later) - no obvious long term solution, but a future version will allow for an easier way to share the cache to make sure others can also run the script instantly - no easy way to "give back" to Wikidata --- # `tidywikidatar` <div style ="float: right;" ><img src="img/tidywikidatar_logo.png" style = "width:256px;"></img></div> Check out website with documentation and examples: https://edjnet.github.io/tidywikidatar/ [](https://cran.r-project.org/package=tidywikidatar) [](https://r-pkg.org/pkg/tidywikidatar) [](https://r-pkg.org/pkg/tidywikidatar) - everything in tabular format - one row, one piece of information - easy local caching - easy integration with `dplyr` piped routines - get image credits from WikiMedia commons - include Wikipedia in the exploration, or use it as a starting point