Skip to content

All your locations are belong to us

A recent push for a UK photography contest reminded me of an issue I have begrudged for a quite a while. On the talk page for that contest, I pointed to several tools of mine, dealing with images and locations. But they only show aspects of those, like “Wikidata items without images”. What about the others? WDQS can show maps of all Wikidata items in a region, but what about Wikipedia? The mobile app can show you things with Wikipedia articles nearby, but what about Commons? I don’t recall a way to see Commons images taken near a location (WD-FIST can find them, but without a map). The data exists, but is either hard to get to, or “siloed” in some tool/app.

Screen Shot 2016-08-20 at 18.06.26Wouldn’t it be great to get a map with all this information on it? All of Wikidata? All of Wikipedia? All of Commons? At once?

What should that look like? The photography contest scenario, and change in general web usage patterns, suggest a strong emphasis on mobile. Which in turn tends to be “no frills”, as in, a focus on what is important: The map, and the objects on it.

So I decided (for the time being) to get rid of the query functions in WD-FIST, and the clutter in WikiShootMe, and start from scratch, with (essentially) just a big map, using the bleeding-edge versions of JS libraries like bootstrap, jQuery, and leaflet. So without further ado, I present WikiShootMe, version 3 (pre-alpha). As it is, the tool defaults to your coordinates, which may be your local hub (as in my case, in the screenshot). There are four layers, which can be individually toggled:

  1. Wikidata items with images (in green)
  2. Wikidata items without image (in red, the Wikipedia will change with your language selection)
  3. Commons images (in blue)
  4. Wikipedia articles (smaller, in yellow, mostly overlapping Wikidata items)

There is also a grey circle in the center, which is your (or your local hub’s) position. On mobile, this should move with you (but I haven’t tested that, as it would require leaving the house). All of these have a pop-up, when you click or touch the circle. It shows the linked title of the object, and, for Wikidata items with images and Commons images, it shows the respective image.

All these data sources will update when you move the map, as well as zoom, up to a certain zoom factor. Below that, an “Update” button will appear to update manually, but it can take a long time, even with the number of objects limited.

Screen Shot 2016-08-20 at 18.46.05I find it amazing how many geo-coded images there are already on Commons (even though the API will only give me 500 at a time). Maybe that is the geograph effect here in the UK, which let to the import of hundreds of thousands of free images to Commons.But I also found a funny pattern in Cologne, Germany, which turned out to be a series of images taken by Wikimedia volunteers from a balloon!

Now, to be extra clever, I tried to add an upload function to the pop-up of Wikidata items without an image. You can select a file from disk, or use the camera as a source on mobile. It will pre-fill the title and the {{Information}} template with a link to the respective Wikidata object. However, several problems occur with that:

  • I only could get the “old” Commons upload page to work with the pre-filled data
  • I could find no documentation on <form> parameters for the Upload wizard
  • I haven’t actually tested if the upload works
  • There seems to be no way to automatically add the uploaded image to the Wikidata item

A way around all that would be to upload the image to the tool itself, then transfer it to Commons via OAuth. This would also allow me to add the new image as the P18 on the Wikidata item. This is an option to be explored, especially if the Upload Wizard remains opaque to me.

Update: I have added OAuth to the tool. Once authorised, you can upload a new image for a Wikidata item from both desktop and mobile (gallery or camera directly) with one click. It fills in file name, coordinates, default license etc. It even adds the image to the item after upload automatically. All this opens in a new tab, on the page for the uploaded image, to give you a chance to add more information.

As usual, I am quite open for bug reports, feature requests (yes, it’s bare-bones at the moment), and technical support by volunteers/WMF.

16 Comments

  1. Jan Ainali wrote:

    Awesome tool! I searched a little in areas that I know of and realized that quite often there are Commons images close to Wikidata items without images. After that you have inspected that these actually match, some way to easily add the image to the Wikidata item (drag-and-drop?) would be really nice.

    Tuesday, August 23, 2016 at 18:32 | Permalink
  2. Magnus wrote:

    Drag’n’drop doesn’t work easily with the map technology, but I have actually planned for this scenario: At the bottom of the pop-up for the image is the full file name. Triple-click it to select it, copy it, then open the pop-up of the item, paste the filename into the box, and click on “Add to item”. Done!

    Tuesday, August 23, 2016 at 22:42 | Permalink
  3. ChristianKl wrote:

    It would be nice to have a straightforward way to create Wikidata objects. I went through the images in Berlin and there are a lot of commemorative plaques that currently don’t have Wikidata objects for their existance.

    Wednesday, August 24, 2016 at 10:53 | Permalink
  4. Magnus wrote:

    There is. Just right-click on the map, enter a label, and an item will be created. It’s described in the manual.

    Wednesday, August 24, 2016 at 11:03 | Permalink
  5. Magnus wrote:

    Or did you mean “directly from the image pop-up”?

    Wednesday, August 24, 2016 at 11:12 | Permalink
  6. Magnus wrote:

    @CHRISTIANKL Done!

    Wednesday, August 24, 2016 at 12:58 | Permalink
  7. Alessandro Marchetti wrote:

    I think that the interaction between wikidata and wikipedia image management is no surprise. It was cited in https://meta.wikimedia.org/wiki/Grants:IEG/Wiki_needs_pictures that you endorsed in November 2015.

    But that interaction was intended to be social not just “coding”. Contact with local communities clearly showed that normal ns-0 users want to use wikidata in everyday’s life if it is proven to be useful. Italian users I’ve contacted one by one have already used the old wikishootme, also WDFIST to start to clean up their areas. That’s why Italy has a now lower density of missing P18 compared to other areas. But the problem was de facto solved with previous tools, just informing the local users. What I realized that was really missing was more a campaign than a “super tool”. So the idea was to create a tool with a campaign, its synergy showing the increase of expertise of the users themselves.

    The use of local image requested categories was not introduced in Wiki needs pictures because it was efficient (we think many of them can be removed, and we said many times) but because it proved this gap to inexpert eyes, that could switch from one obsolete system they were more used to a partial yet multilayer system.

    But, as with WDFIST, you can’t just give a “bot” if people don’t know what it actually automatizes. A decent tool for the mass is much more effective at a wiki level than an advanced tool few people can really understand.

    I used the old wikishootme to teach a basis of wikidata and images, and it was totally ok (without query functions, no newbie liked those). This tool v3 simply can’t be given to a “nerd-newbie”. There’s too much I have to say.

    And also if you show all available pictures on panoramio or whatever, they usually fit to existing images or articles, not often in missing one. Uk is a special case and we still have thousands of pictures imported in the early 2010s to be revised on commons. This shows that we lack manpower to deal with automation, and semi-automation might not be the perfect answer, if it opens even more fronts.

    I say that because I have done so many tests with missing images. Our problem are now “usually isolated items” that can be taken into account especially by people who know the areas. And also have the expertise to take a pictures whilst they’re there to objects who don’t have an article yet but they know thy will, because they’re the future authors for example. This is particularly true in isolated countries linked to language edition of wikipedia not fully developped. It’s a “game” of diffuse expertise, not of a mass game. The “mass semiautomated approach” is the same that lead few users to deal with the update of P18 on wikidata in the first place and it’s not complete. It’s a fact, it was not when we started in 2015 to discuss with local users, and it is not now yet. It took years to associate images to wikidata item, now you think about “create Wikidata objects” from images. Do you see how a big risk it is at this rhythm?

    Also, I see from the map, just because I know the network of ns0 article of my areas, that some “New Wikidata item based on this image” will crate duplicates. More work for the nerds, but I swear: regular user who write article, the dozens I spoke with, don’t want that. Statistically it would be the “nerd” to find it and use it before the “robust ns0 contributor”.

    In summary, the challenge I saw talking to users is more like a quality search and that needs calm. It needs to teach people to do good quality steps, and that’s something you learn slowly first.

    For example, first they learn how to reorganize commons, them how do a wikidata item, etc. For these users, if you talk to them, “automatic” upload by people who might not know the areas very well are a source of irritation, a lot of work to do because they are precise with details, that’s what they need when they write articles. Our main goal should be not to burn those users, and let them to learn and reach high quality standards.

    This tool could be ok once the situation is stationary (which is not) and the medium users who really edit ns0 are more expert (which are not).

    Thursday, August 25, 2016 at 14:24 | Permalink
  8. Magnus wrote:

    Hi Alessandro,
    wow, long text. I take it the upshot is that you think this version of WikiShootMe is unsuitable for newbies.
    I disagree, considering the inherent complexity of the matter. IMHO the new version does everything the old one did, plus “X”, in a much more intuitive way. (The old versions are still working, and linked from the “About” page, if you care.)
    You are welcome to improve the manual, so that the tool easier to understand for newbies.
    I also did not attempt to undermine the IEG; it is, in fact, completely unrelated. As usual, I built things for myself, and then let people play with it.
    Letting people create Wikidata items from images is, of course, a risk. So is letting people edit Wikipedia. Who knows what nonsense they will write! Jesting aside, this is actually a good functionality for the “expert photographers” you mentioned, who will take a strategic surplus of pictures and upload them. These can now be turned into Wikidata items easily, and thereby also “anchor” the image for future Wikipedia articles.
    I am, however, at a loss as to what you want from me. Something “social”, apparently. I don’t really do social 😉 I wrote a tool, and you say “it’s bad, because it doesn’t solve some other problem I care about”. The sound you’re hearing is me shrugging violently.

    Thursday, August 25, 2016 at 16:37 | Permalink
  9. Alessandro Marchetti wrote:

    What does it mean “shrugging violently”?

    Look, this is going to be long, maybe you don’t care, and I will write it later on meta, but I’m here now so I hope you don’t mind if I finish the discussion. I will use your comment to revise it later.

    I get that you do tool for yourself, it is clear, since the beginning. But tools once introduced in a wiki ecosystem are not neutral. The “patterns” they introduce affect the pattern of editors, their choices, and also in turn the evolution of some aspects of the platforms.

    If you don’t make tools for people, you have to “make people for tools” to get a balanced wiki environment and that takes time.

    The “I care about” is what people’ve talked with care about. I care about helping them and their work, ns0 work. The amount of users (decent users with kb of edits) who know wikidata is low, something like 5-10% of those I’ve contacted so far. Almost noone knows usually flinfo, so they don’t update from flickr for example, or panoramio or whatever. This is the starting point. I showed to some of them your version 3 already, right this afternoon, and for example for certain areas it was not a succesful reaction. It’s like you click on 4-5 pictures and noone is right and you’re disappointed. Amongst the tests I also did, the only area so far where the tool was smoothly ok are the USA, but mainly because there’s a terrible P18 backlog there. I suspect that has to do with countries where wikidata was more or less popular so far. But again it isn’t a new tool that solves the backlog, it is just the manpower. You could explain to a group of additional USA users WDFIST and they would do most of that ncssary work in any case.

    Let’s talk about how the tool affect the quality. Because it was in areas with high level of ns0 content quality on local wikis that this tool “struggled”. If you have a missing image on wikidata or wiki but there’s something not great or partially overlapping or not perfect quality on commons or an external archive, do you use it? These types of tools increase the semiautomation that forces to fill the gap with what you have. Even if you don’t accept a picture, someone else might do after a while. For example if I have a picture of a church with just one building and the P18 of the item of a hamlet I might say that’s not the full hamlet, I might know better pictures are possible of that hamlet and not use it on wikidata. But what about the next user? statistically, the more he’s playing a “P18 game”, the more he will just use it. This choice than propagates to other local wikis where it will take time to change that pictures once a better one is available. In addition to that, filling the gap drives the attention somewhere else, where a gap is still marked on the map. But actually, it is much better to discard an image that is 70% right (like too partial for example) and send a ns-0 oriented user in the area with camera.(s)he can start for what is clearly missing and follow his/her expertise:(s)he knows the details, (s)he understands that the object is important even if it’s not on wikidata or the local wikipedia yet, (s)he knows which specific details of it are really necessary. A little bit later (just few months if you play it right), you get a much better results. To be honest, I prefer (s)he comes to a point on the map and do a good job than just avoiding it because someone who never saw the place, upload there a not 100% correct image because it was the only one available during a “nerdy” afternoon session. You get what I mean?

    Statistically, this tool cannot be played by standard ns0 contributors (just talk to them, assess their level), but it will played more by users that fell the P18 insertion as a formal more than substantial step. This means you are not actually speeding up the search of the “final” pictures you will use, and by that I mean the type of good, detailed pictures we need for articles nowadays at least in some wikipedias.

    That’s the point: Wikidata will work without P18, and it did without it so far. it’s not a real structured data, it’s for “the pleasure of the eye”. So there’s no hurry unless you want to be sure that the next bot who create thousands of “low-quality” articles will have “its own picture from wikidata”. I think we can survive this occurrence. But I would prefer to see those P18 used as a learning device for a really expert, content-oriented user, because it’s a really good opportunity.

    BTW, you didn’t undermine the IEG. The other tool will be basically completed, delays are related to offwiki reasons, and it was different. Most of all I encourage plurality. My worries are for the final quality here. Both tools recognize that wikipedia and wikidata need to be more related on the images issue. But again, that wasn’t intended as a “game to fill P18”. It is an attempt to create a stronger, quality oriented expertise. The “local” categories of missing images are in fact a backup plan. When a mediocre P18 image will be uploaded in any case, they can be used to point out that something needs further attention. But local communities still need time to process this dimension, it’s a possible evolution that will take some years.

    Finally, creating items from images directly is a big deal. First of all, it could make difficult for newbies to understand the structured data part. But the core problem is that people have much lower wikidata-literacy than wikipedia. This may create duplicates and a lot of work. Just to give you the idea I wanted to inform the main it-N users creators of not wikidata-connected wikipedia articles about how to create a new item. I was stopped, this was considered “too much for them”. So I am not the most critical person you can find on the issue.

    This tool will be fit great in few years, it is early now. Come on, even when I was teaching how to insert P18 some wikidata main users grumbled that they were scared they had too much wrong revisions to check or that was wrong for users who knew nothing of wikidata.

    Friday, August 26, 2016 at 02:06 | Permalink
  10. Gustav wrote:

    Hi.Great tool. Very useful to see images from commons, articles and wd-items on the same map.

    I have a couple of questions. Could it be a bit harder to add a new object. I do often accidentally right-click. If a click outside the box would make it disappear or if there was a step making a popup before the box appears it would be great.

    The second question is ifyou could add a possibility to filter wd-items on P31. Some types are more interesting than others. Preferably with subclasses.

    Lastly. Is it possible to show the images from commons differently. If there is a area with a lot of images it completely dominates making other images not appear. I would like to prioritize images far from other images or something.

    Friday, August 26, 2016 at 10:50 | Permalink
  11. Magnus wrote:

    @GUSTAV I have thrown in a context menu, that should take care of your first issue, and allow for more location-specific functions in the future.
    A P31 filter is on my to-do-list, but the interface is a little tricky. Ideally, I want to be able to mix user-defined SPARQL in there, but not make it too complicated either.
    Not sure how to make Commons images “differently”, or what exactly you mean by that. Is the problem that multiple Commons images can be “on top” of each other in some areas? I could try adding some position jitter, maybe?

    Friday, August 26, 2016 at 11:38 | Permalink
  12. Gustav wrote:

    Great! That was fast.
    Regarding commons, the problem can be seen here: https://tools.wmflabs.org/wikishootme/#lat=57.68045887950784&lng=11.95518493652344&zoom=16&layers=commons
    There is a botanical garden there where a lot of images have been shot. There are a lot outside as well but since there are that many inside the garden they completely dominate.

    Friday, August 26, 2016 at 11:56 | Permalink
  13. Magnus wrote:

    @GUSTAV SPARQL filter implemented, example:
    https://tools.wmflabs.org/wikishootme/#lat=51.985488282603676&lng=0.32066345214843756&zoom=11&sparql_filter=%3Fq%20wdt%3AP31%7Cwdt%3AP279*%20wd%3AQ55488&layers=wikidata_image,wikidata_no_image

    I’m too lazy to write a manual entry right now, but feel free 😉

    Not sure what to do about the Commons images; the effect does largely go away when you zoom in. More transparency? Heatmap?

    Friday, August 26, 2016 at 14:06 | Permalink
  14. Gustav wrote:

    Looks really good. Shouldn’t the vertical bar in the SPARQL filter be changed to a forward slash? As it is now this query:
    ?q wdt:P31|wdt:P279* wd:Q41176.
    MINUS { ?q wdt:P31 wd:Q41176. }
    gives no results. But with a forward slash it does (as it should).
    Regarding the commons images: I don’t know how to solve it. The two obvious solutions are to zoom in until there are fewer images on screen or to make the maximum number of images higher so all are shown. Neither is a good solution.
    I am not very familiar with relational databases but some ordering of the results might help (but I don’t know any that does). However as the centermost images are shown first it might be ok without doing anything.
    The ugly solution that does work is to divide the screen in segments and query each one separately…

    Saturday, August 27, 2016 at 10:16 | Permalink
  15. Gustav wrote:

    Thks for the fix.
    One more suggestion. Would it be possible to show any label if none exists on the selected language. Many names are similar and I think anything is more helpful than the Q-number.

    Monday, September 5, 2016 at 18:48 | Permalink
  16. Magnus wrote:

    @Gustav I actually had that in place until today, but it turns out that SERVICE-based labels take much longer than rdsf:label based ones. As in, timeout-longer.

    Monday, September 5, 2016 at 23:01 | Permalink